Sep 09 20:52:32.733043 ip-10-0-142-161 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:33.077420 ip-10-0-142-161 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:33.077420 ip-10-0-142-161 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:33.077420 ip-10-0-142-161 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:33.077420 ip-10-0-142-161 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:33.077420 ip-10-0-142-161 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:33.079442 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.079360    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:33.082427 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082404    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.082427 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082421    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.082427 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082426    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.082427 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082431    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.082427 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082436    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082439    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082445    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082449    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082453    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082456    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082460    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082464    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082467    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082471    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082475    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082479    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082482    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082486    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082490    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082493    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082497    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082501    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082505    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082509    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.082709 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082512    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082516    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082520    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082524    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082527    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082530    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082534    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082538    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082542    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082545    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082550    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082554    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082558    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082563    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082568    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082573    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082577    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082581    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082585    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082589    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.083580 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082593    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082597    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082601    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082605    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082609    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082615    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082622    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082628    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082633    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082638    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082642    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082646    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082650    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082654    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082657    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082661    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082665    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082669    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082673    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082677    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.084346 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082681    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082685    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082690    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082694    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082698    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082702    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082708    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082712    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082716    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082722    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082731    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082737    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082761    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082766    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082770    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082775    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082779    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082783    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082787    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.084921 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082791    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082795    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082799    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082803    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082808    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082811    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082815    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082819    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082824    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082828    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082832    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082836    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082840    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082843    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082847    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.082852    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084776    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084790    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084794    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084796    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.085387 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084799    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084802    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084806    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084809    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084811    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084814    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084817    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084820    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084823    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084825    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084828    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084830    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084833    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084838    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084841    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084844    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084847    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084850    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084852    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.085887 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084855    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084858    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084860    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084863    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084865    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084868    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084871    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084874    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084877    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084879    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084883    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084886    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084889    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084893    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084897    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084901    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084903    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084906    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084909    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084911    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.086369 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084914    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084917    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084919    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084922    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084924    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084927    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084930    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084932    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084935    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084937    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084940    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084942    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084945    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084947    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084950    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084952    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084955    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084957    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084960    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084962    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.086888 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084964    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084967    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084970    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084973    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084975    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084978    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084980    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084983    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084985    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084987    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084990    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084992    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084995    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084997    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.084999    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085002    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085004    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085007    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085009    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085011    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.087384 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085016    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085020    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085024    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085027    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085030    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085032    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085035    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085039    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085041    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085044    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085046    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085048    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085051    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085053    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085057    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085060    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085460    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085465    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085468    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.087881 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085472    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085476    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085479    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085482    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085485    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085487    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085490    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085492    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085495    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085498    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085500    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085503    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085505    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085508    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085511    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085513    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085516    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085518    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085520    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085523    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.088350 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085525    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085528    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085530    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085532    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085535    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085538    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085540    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085543    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085546    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085548    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085551    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085553    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085556    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085558    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085561    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085564    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085566    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085569    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085571    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085573    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.088895 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085576    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085579    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085581    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085583    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085587    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085590    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085594    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085597    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085599    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085602    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085605    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085609    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085611    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085614    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085617    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085620    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085622    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085625    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085627    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.089368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085631    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085633    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085635    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085638    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085640    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085644    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085647    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085650    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085652    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085655    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085657    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085659    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085662    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085664    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085666    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085669    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085671    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085673    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085676    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085678    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.089850 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085681    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085684    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085686    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085688    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085691    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085693    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085695    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085699    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085702    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085704    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085707    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085709    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085712    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085715    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085717    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085720    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.085722    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085812    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085849    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085860    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085865    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:33.090368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085869    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085872    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085876    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085880    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085883    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085886    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085889    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085892    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085898    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085901    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085904    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085906    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085910    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085913    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085917    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085921    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085924    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085927    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085930    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085933    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085936    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085939    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085942    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085945    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:33.091000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085948    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085950    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085953    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085956    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085960    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085964    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085974    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085978    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085981    2578 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085984    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085989    2578 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085992    2578 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085995    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.085998    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086001    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086009    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086015    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086018    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086020    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086023    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086026    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086029    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086032    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086037    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086039    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:33.091568 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086042    2578 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086046    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086048    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086051    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086054    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086057    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086060    2578 flags.go:64] FLAG: --help="false"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086063    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086066    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086068    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086071    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086074    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086078    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086080    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086083    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086086    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086088    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086091    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086094    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086097    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086100    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086102    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086105    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086109    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:33.092211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086112    2578 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086114    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086117    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086120    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086125    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086129    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086132    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086135    2578 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086138    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086141    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086144    2578 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086147    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086151    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086154    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086157    2578 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086160    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086163    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086166    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086168    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086171    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086174    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086177    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086183    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086186    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086189    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:33.092813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086192    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086195    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086201    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086203    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086206    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086209    2578 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086213    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086216    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08776eeb27cce7c37"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086219    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086221    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086224    2578 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086227    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086230    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086233    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086237    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086240    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086243    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086246    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086249    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086252    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086255    2578 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086257    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086260    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086263    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086266    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:33.093439 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086269    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086271    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086274    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086277    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086279    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086282    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086285    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086288    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086291    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086293    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086296    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086301    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086303    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086307    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086311    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086313    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086316    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086319    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086321    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086324    2578 flags.go:64] FLAG: --v="2"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086327    2578 flags.go:64] FLAG: --version="false"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086333    2578 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086338    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.086342    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:33.094102 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086456    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086461    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086463    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086466    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086469    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086471    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086474    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086476    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086480    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086484    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086487    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086490    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086492    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086495    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086497    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086500    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086505    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086507    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.094671 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086510    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086512    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086515    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086517    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086520    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086522    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086525    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086527    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086530    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086532    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086536    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086540    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086542    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086545    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086547    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086550    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086552    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086554    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086557    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086559    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.095156 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086562    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086564    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086566    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086569    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086573    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086576    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086579    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086582    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086584    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086586    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086589    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086592    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086594    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086597    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086600    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086602    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086604    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086607    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086609    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.095647 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086612    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086615    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086617    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086621    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086623    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086626    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086629    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086631    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086634    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086636    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086639    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086641    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086644    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086646    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086648    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086651    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086653    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086656    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086658    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086661    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.096118 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086663    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086665    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086668    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086670    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086672    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086675    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086679    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086682    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086685    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086688    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086691    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086694    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086696    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086699    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086702    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086706    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086708    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086711    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086714    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086717    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.096602 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.086719    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.087181    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.094277    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.094293    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.094310    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.094319    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094370    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094375    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094379    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094382    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094385    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094388    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094390    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094393    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094396    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.097099 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094399    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094402    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094404    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094407    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094411    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094415    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094417    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094420    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094423    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094426    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094429    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094431    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094434    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094437    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094439    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094442    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094444    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094447    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094450    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.097533 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094453    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094456    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094459    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094461    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094464    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094466    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094469    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094471    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094474    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094476    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094479    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094481    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094484    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094487    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094489    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094492    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094494    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094498    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094501    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.098024 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094504    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094507    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094510    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094512    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094515    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094517    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094520    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094522    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094525    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094527    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094530    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094533    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094535    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094539    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094542    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094545    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094547    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094550    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094552    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094556    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.098517 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094559    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094562    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094564    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094567    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094570    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094572    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094575    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094577    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094580    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094582    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094585    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094587    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094590    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094592    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094594    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094597    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094599    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094602    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094604    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094607    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.099122 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094609    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094611    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094614    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094617    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094620    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094623    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094627    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094629    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094632    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094634    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094637    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094640    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.094645    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094766    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094771    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:33.099599 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094775    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094778    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094781    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094783    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094786    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094789    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094791    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094794    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094797    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094799    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094802    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094804    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094807    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094809    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094812    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094815    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094817    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094820    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094822    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:33.099982 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094824    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094826    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094829    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094833    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094835    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094839    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094843    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094846    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094849    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094852    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094855    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094858    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094861    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094863    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094866    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094868    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094870    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094873    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094875    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:33.100469 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094878    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094880    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094882    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094885    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094887    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094890    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094892    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094894    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094897    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094899    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094902    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094904    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094906    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094909    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094911    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094914    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094917    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094920    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094922    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094925    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:33.100958 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094927    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094930    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094934    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094937    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094940    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094943    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094946    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094949    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094951    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094954    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094956    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094959    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094961    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094964    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094966    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094968    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094971    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094973    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094975    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094978    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:33.101436 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094981    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094983    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094986    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094988    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094990    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094993    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094995    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.094997    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095001    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095004    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095006    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095009    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095011    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095013    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095016    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095018    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095021    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095023    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:33.101928 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:33.095026    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.095030    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.095120    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.097244    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.097997    2578 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.098090    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.098118    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.100169    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:33.102360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.101225    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:33.112495 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.112477    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:33.116480 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.116462    2578 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:33.117469 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.117374    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:33.122425 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.122404    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 807f1c5b-8109-4ca0-a95c-567c0905516c:/dev/nvme0n1p3 c1fbf0f0-e086-4360-b947-6de7dbed10a2:/dev/nvme0n1p4]
Sep 09 20:52:33.122488 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.122423    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:33.126761 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.126728    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:33.127601 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.127478    2578 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:33.125829606 +0000 UTC m=+0.291156866 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112939 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e16bc3b83a0be67230d3d9921d52d SystemUUID:ec2e16bc-3b83-a0be-6723-0d3d9921d52d BootID:7b2c39b2-24e1-4d99-8524-cfabb20f455a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ea:0d:aa:7a:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ea:0d:aa:7a:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:72:89:4d:91:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:33.127601 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.127601    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:33.127771 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.127671    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:33.128675 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.128653    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:33.128824 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.128677    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:33.129437 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.129427    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:33.129470 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.129440    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:33.129470 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.129447    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:33.130127 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.130117    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:33.131163 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.131153    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:33.135339 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135327    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:33.135442 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135432    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:33.135479 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135461    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:33.135479 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135471    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:33.135535 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135481    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:33.135535 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135490    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:33.135535 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135498    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:33.135829 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135818    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:33.135869 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.135836    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:33.137684 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.137670    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:33.140143 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.140130    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:33.140192 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.140159    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:33.141137 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141125    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141142    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141147    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141152    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141160    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141165    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141178    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141182    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:33.141189 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141189    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:33.141390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141194    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:33.141390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141203    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:33.141390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141209    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:33.141845 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141836    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:33.141845 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.141845    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:33.145638 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.145625    2578 server.go:1268] "Started kubelet"
Sep 09 20:52:33.145768 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.145710    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:33.145822 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.145785    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:33.146018 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.145999    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:33.146362 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.146303    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:33.146466 ip-10-0-142-161 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:33.147278 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.147060    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:33.147278 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.147109    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:33.147455 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.147357    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:33.147978 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.147964    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:33.150642 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.150625    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:33.153335 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.153316    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:33.153429 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.152563    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-161.ec2.internal.18d3c2565afb0fdd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-161.ec2.internal,UID:ip-10-0-142-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-161.ec2.internal,},FirstTimestamp:2026-09-09 20:52:33.145597917 +0000 UTC m=+0.310925176,LastTimestamp:2026-09-09 20:52:33.145597917 +0000 UTC m=+0.310925176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-161.ec2.internal,}"
Sep 09 20:52:33.153729 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.153710    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:33.154092 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.154065    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.154706 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154536    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:33.154706 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154616    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:33.155032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154816    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:33.155032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154825    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:33.155032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154933    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:33.155032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154942    2578 factory.go:55] Registering systemd factory
Sep 09 20:52:33.155032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.154963    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:33.155460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.155442    2578 factory.go:153] Registering CRI-O factory
Sep 09 20:52:33.155460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.155462    2578 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:33.155584 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.155536    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:33.155584 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.155554    2578 factory.go:103] Registering Raw factory
Sep 09 20:52:33.155584 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.155569    2578 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:33.156457 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.156426    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xfl2x"
Sep 09 20:52:33.156622 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.156607    2578 manager.go:349] Starting recovery of all containers
Sep 09 20:52:33.157683 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.157628    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:33.157869 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.157841    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:33.165002 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.164853    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xfl2x"
Sep 09 20:52:33.167966 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.167949    2578 manager.go:354] Recovery completed
Sep 09 20:52:33.169487 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.169426    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:52:33.172331 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.172318    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.174184 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174167    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.174255 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174200    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.174255 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174214    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.174820 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174807    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:33.174820 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174820    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:33.174923 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174835    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:33.174923 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.174845    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.176626 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.176557    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-161.ec2.internal.18d3c2565caf3cbc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-161.ec2.internal,UID:ip-10-0-142-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-161.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-161.ec2.internal,},FirstTimestamp:2026-09-09 20:52:33.1741831 +0000 UTC m=+0.339510360,LastTimestamp:2026-09-09 20:52:33.1741831 +0000 UTC m=+0.339510360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-161.ec2.internal,}"
Sep 09 20:52:33.176856 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.176842    2578 policy_none.go:49] "None policy: Start"
Sep 09 20:52:33.176930 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.176859    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:33.176930 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.176870    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:33.177252 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.177238    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.177292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.177262    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.177292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.177271    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.178881 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.178870    2578 policy_none.go:47] "Start"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.214931    2578 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.214964    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.214974    2578 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.215274    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.215284    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.215384    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.215456    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.215465    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.216015    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:33.229440 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.216057    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.275194 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.275167    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:33.276473 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.276458    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:33.276567 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.276510    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:33.276567 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.276537    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:33.276667 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.276578    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:33.279248 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.279231    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:33.315532 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.315516    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.316300 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.316284    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.316366 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.316311    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.316366 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.316320    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.316366 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.316340    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.323032 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.323011    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.323032 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.323031    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-161.ec2.internal\": node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.342401 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.342338    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.377520 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.377480    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"]
Sep 09 20:52:33.377572 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.377556    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.378319 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.378302    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.378379 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.378332    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.378379 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.378343    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.380268 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.380257    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.380424 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.380413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.380457 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.380438    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.381776 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381758    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.381877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381760    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.381877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381801    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.381877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381806    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.381877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381812    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.381877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.381816    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.383841 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.383828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.383914 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.383849    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:33.384563 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.384545    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:33.384641 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.384577    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:33.384641 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.384592    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:33.425202 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.425178    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-161.ec2.internal\" not found" node="ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.429401 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.429385    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-161.ec2.internal\" not found" node="ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.442638 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.442623    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.457031 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.457013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a96d69af6916da553cbc5c637b97bef-config\") pod \"kube-apiserver-proxy-ip-10-0-142-161.ec2.internal\" (UID: \"3a96d69af6916da553cbc5c637b97bef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.457105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.457032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.457105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.457049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.542889 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.542868    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.557297 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.557364 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.557364 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a96d69af6916da553cbc5c637b97bef-config\") pod \"kube-apiserver-proxy-ip-10-0-142-161.ec2.internal\" (UID: \"3a96d69af6916da553cbc5c637b97bef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.557460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.557460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17e73fc6365d65c00c868dbb423c770e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal\" (UID: \"17e73fc6365d65c00c868dbb423c770e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.557460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.557439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a96d69af6916da553cbc5c637b97bef-config\") pod \"kube-apiserver-proxy-ip-10-0-142-161.ec2.internal\" (UID: \"3a96d69af6916da553cbc5c637b97bef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.643804 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.643706    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.728242 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.728215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.733204 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:33.733186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:33.744688 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.744673    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.845265 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.845232    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:33.945855 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:33.945784    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.046436 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:34.046402    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.092445 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.092418    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:34.098522 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.098504    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:34.098631 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.098614    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:34.098672 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.098615    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:34.147081 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:34.147054    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.171374 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.171337    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:33 +0000 UTC" deadline="2028-03-16 12:25:16.13950456 +0000 UTC"
Sep 09 20:52:34.171374 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.171364    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13287h32m41.968144884s"
Sep 09 20:52:34.248013 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:34.247949    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.309553 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:34.309519    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e73fc6365d65c00c868dbb423c770e.slice/crio-ce0490442f95064ccabcddab56d624cebe1ce8b882f9b5cd59d6993be2048960 WatchSource:0}: Error finding container ce0490442f95064ccabcddab56d624cebe1ce8b882f9b5cd59d6993be2048960: Status 404 returned error can't find the container with id ce0490442f95064ccabcddab56d624cebe1ce8b882f9b5cd59d6993be2048960
Sep 09 20:52:34.310177 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:34.310159    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a96d69af6916da553cbc5c637b97bef.slice/crio-1b28fd388473e4bfce8161a8d41a2271620eb6cad892d90f07c3d48a19d3576e WatchSource:0}: Error finding container 1b28fd388473e4bfce8161a8d41a2271620eb6cad892d90f07c3d48a19d3576e: Status 404 returned error can't find the container with id 1b28fd388473e4bfce8161a8d41a2271620eb6cad892d90f07c3d48a19d3576e
Sep 09 20:52:34.313943 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.313925    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:34.348797 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:34.348777    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.449310 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:34.449285    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-161.ec2.internal\" not found"
Sep 09 20:52:34.501997 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.501934    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:34.554062 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.554038    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:34.567955 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.567939    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:34.569485 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.569472    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal"
Sep 09 20:52:34.581063 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.581048    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:34.735709 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:34.735687    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:35.043877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.043842    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:35.136097 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.136010    2578 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:35.143686 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.143663    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:35.144191 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.144163    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4","openshift-cluster-node-tuning-operator/tuned-s886d","openshift-image-registry/node-ca-c7cjx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-vfjds","openshift-network-diagnostics/network-check-target-9g5pw","kube-system/global-pull-secret-syncer-4wzrr","openshift-dns/node-resolver-qdng8","openshift-multus/multus-v2qvz","openshift-multus/network-metrics-daemon-5k56s","openshift-network-operator/iptables-alerter-g64bs","openshift-ovn-kubernetes/ovnkube-node-dv7rn","kube-system/konnectivity-agent-ldv7d"]
Sep 09 20:52:35.146890 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.146865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:35.146989 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.146944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:35.149569 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.149548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.149685 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.149624    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:35.152337 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.152315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.154259 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.154235    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:35.154352 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.154258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.155292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.155270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.155772 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.155747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z4gpk\""
Sep 09 20:52:35.157716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.157716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.157716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.157716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157551    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhx7d\""
Sep 09 20:52:35.157992 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157863    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:35.157992 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.157987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.160287 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.160256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-prc5b\""
Sep 09 20:52:35.160397 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.160301    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:35.160947 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.160608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:35.161277 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.161072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.164132 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.164319 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rt9mm\""
Sep 09 20:52:35.164442 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:35.164518 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:35.164591 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:35.164788 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.164889 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.164805    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.166443 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.166422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.167118 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1c247cc-2776-45c8-9fae-f0374394b7a7-agent-certs\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.167211 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.167273 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-kubernetes\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167273 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-run\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167273 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-lib-modules\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167413 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-host\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167413 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02664476-02ce-4f9e-9309-b18a425e76c6-serviceca\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.167413 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bt4\" (UniqueName: \"kubernetes.io/projected/02664476-02ce-4f9e-9309-b18a425e76c6-kube-api-access-j8bt4\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1c247cc-2776-45c8-9fae-f0374394b7a7-konnectivity-ca\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysconfig\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-tuned\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-tmp\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02664476-02ce-4f9e-9309-b18a425e76c6-host\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.167560 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-modprobe-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-systemd\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-sys\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96q4\" (UniqueName: \"kubernetes.io/projected/ee1b4883-3bbb-459e-b700-5a38c2131ba3-kube-api-access-g96q4\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-conf\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-var-lib-kubelet\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.167935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.167779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2wh\" (UniqueName: \"kubernetes.io/projected/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-kube-api-access-nt2wh\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.168280 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-82cx7\""
Sep 09 20:52:35.168280 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.168381 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168346    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.168665 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168612    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:35.169260 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.169260 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.168934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbnxx\""
Sep 09 20:52:35.169415 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.169390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.169800 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.169536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.169800 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.169670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:35.170979 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.170683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:35.170979 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.170738    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:35.170979 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.170822    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:35.171137 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.170990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.171542 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.171459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9sfcq\""
Sep 09 20:52:35.171542 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.171491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.171809 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.171776    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:35.172132 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.172111    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.173733 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.173702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:35.174087 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.174069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4q2w5\""
Sep 09 20:52:35.174363 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.174346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.174440 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.174420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:35.176718 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.176697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.179409 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.179389    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kh2p6\""
Sep 09 20:52:35.179626 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.179609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:35.179862 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.179840    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:35.180604 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.180321    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:35.203360 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.203338    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-46xg6"
Sep 09 20:52:35.212907 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.212879    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-46xg6"
Sep 09 20:52:35.256022 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.255993    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:35.268712 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41323f0f-2da9-4852-bb4a-92ea2f770f87-iptables-alerter-script\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.268865 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.268865 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.268865 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-bin\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.268865 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-netns\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.268865 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-hostroot\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-daemon-config\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-device-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-sys-fs\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02664476-02ce-4f9e-9309-b18a425e76c6-serviceca\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bt4\" (UniqueName: \"kubernetes.io/projected/02664476-02ce-4f9e-9309-b18a425e76c6-kube-api-access-j8bt4\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.268999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-etc-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269052 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-kubelet\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-conf-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86wc\" (UniqueName: \"kubernetes.io/projected/41323f0f-2da9-4852-bb4a-92ea2f770f87-kube-api-access-r86wc\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-netns\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-socket-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-modprobe-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-node-log\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-env-overrides\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96q4\" (UniqueName: \"kubernetes.io/projected/ee1b4883-3bbb-459e-b700-5a38c2131ba3-kube-api-access-g96q4\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-cnibin\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-tmp-dir\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.269328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-systemd-units\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-kubelet-config\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-dbus\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1c247cc-2776-45c8-9fae-f0374394b7a7-agent-certs\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-kubelet\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-systemd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlq2\" (UniqueName: \"kubernetes.io/projected/0861aad9-818e-46fb-bef9-06e1d0d99c6b-kube-api-access-5vlq2\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbvj\" (UniqueName: \"kubernetes.io/projected/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-kube-api-access-gwbvj\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-netd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-system-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-socket-dir-parent\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.269933 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-registration-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41323f0f-2da9-4852-bb4a-92ea2f770f87-host-slash\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-sys\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-script-lib\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-sys\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cni-binary-copy\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.269995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-modprobe-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-multus-certs\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-d\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4h6\" (UniqueName: \"kubernetes.io/projected/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kube-api-access-ng4h6\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-var-lib-kubelet\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-etc-kubernetes\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-lib-modules\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02664476-02ce-4f9e-9309-b18a425e76c6-serviceca\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-var-lib-kubelet\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1c247cc-2776-45c8-9fae-f0374394b7a7-konnectivity-ca\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.270738 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-tuned\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-slash\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-lib-modules\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-k8s-cni-cncf-io\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270525    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-systemd\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-conf\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-systemd\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2wh\" (UniqueName: \"kubernetes.io/projected/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-kube-api-access-nt2wh\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-ovn\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1c247cc-2776-45c8-9fae-f0374394b7a7-konnectivity-ca\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97hq\" (UniqueName: \"kubernetes.io/projected/2ce78ece-1667-4ec6-be73-9f59da6e3b72-kube-api-access-d97hq\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysctl-conf\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-kubernetes\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.270963    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.270990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-run\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.271609 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.271033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:35.771009909 +0000 UTC m=+2.936337158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-host\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-kubernetes\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-var-lib-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-run\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysconfig\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-tmp\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-host\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-sysconfig\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02664476-02ce-4f9e-9309-b18a425e76c6-host\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-system-cni-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-hosts-file\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02664476-02ce-4f9e-9309-b18a425e76c6-host\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-log-socket\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cnibin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-bin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.272476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9jf\" (UniqueName: \"kubernetes.io/projected/84aa425b-634f-42cb-8758-a6980a14f99e-kube-api-access-gk9jf\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.273315 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-config\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.273315 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.271573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-multus\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.274776 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.274751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-etc-tuned\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.275249 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.275210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-tmp\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.275249 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.275250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1c247cc-2776-45c8-9fae-f0374394b7a7-agent-certs\") pod \"konnectivity-agent-ldv7d\" (UID: \"e1c247cc-2776-45c8-9fae-f0374394b7a7\") " pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.282020 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.281939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal" event={"ID":"17e73fc6365d65c00c868dbb423c770e","Type":"ContainerStarted","Data":"ce0490442f95064ccabcddab56d624cebe1ce8b882f9b5cd59d6993be2048960"}
Sep 09 20:52:35.282341 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.282277    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:35.282341 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.282304    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:35.282341 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.282318    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:35.282694 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.282392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:35.782372258 +0000 UTC m=+2.947699533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:35.283598 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.283519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal" event={"ID":"3a96d69af6916da553cbc5c637b97bef","Type":"ContainerStarted","Data":"1b28fd388473e4bfce8161a8d41a2271620eb6cad892d90f07c3d48a19d3576e"}
Sep 09 20:52:35.285161 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.285116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96q4\" (UniqueName: \"kubernetes.io/projected/ee1b4883-3bbb-459e-b700-5a38c2131ba3-kube-api-access-g96q4\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.285634 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.285609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2wh\" (UniqueName: \"kubernetes.io/projected/cb1219fa-b16b-4c7b-96f3-dea0ed0d3381-kube-api-access-nt2wh\") pod \"tuned-s886d\" (UID: \"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381\") " pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.286462 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.286442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bt4\" (UniqueName: \"kubernetes.io/projected/02664476-02ce-4f9e-9309-b18a425e76c6-kube-api-access-j8bt4\") pod \"node-ca-c7cjx\" (UID: \"02664476-02ce-4f9e-9309-b18a425e76c6\") " pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.372851 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-slash\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.372851 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-k8s-cni-cncf-io\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-ovn\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97hq\" (UniqueName: \"kubernetes.io/projected/2ce78ece-1667-4ec6-be73-9f59da6e3b72-kube-api-access-d97hq\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-var-lib-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-slash\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-system-cni-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-var-lib-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-k8s-cni-cncf-io\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-hosts-file\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.373058 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-log-socket\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-system-cni-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cnibin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-bin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-hosts-file\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-log-socket\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.372949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-ovn\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-bin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cnibin\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9jf\" (UniqueName: \"kubernetes.io/projected/84aa425b-634f-42cb-8758-a6980a14f99e-kube-api-access-gk9jf\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-config\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-multus\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41323f0f-2da9-4852-bb4a-92ea2f770f87-iptables-alerter-script\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-cni-multus\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-bin\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.373423 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-netns\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-hostroot\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-daemon-config\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-device-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-sys-fs\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-bin\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-hostroot\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-etc-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-netns\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-etc-openvswitch\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-device-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-kubelet\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-conf-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-sys-fs\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.374345 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-var-lib-kubelet\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86wc\" (UniqueName: \"kubernetes.io/projected/41323f0f-2da9-4852-bb4a-92ea2f770f87-kube-api-access-r86wc\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-netns\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-conf-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-socket-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-node-log\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-env-overrides\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-cnibin\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-tmp-dir\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-systemd-units\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-kubelet-config\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-dbus\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41323f0f-2da9-4852-bb4a-92ea2f770f87-iptables-alerter-script\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.375176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.373983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-kubelet\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-systemd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-config\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlq2\" (UniqueName: \"kubernetes.io/projected/0861aad9-818e-46fb-bef9-06e1d0d99c6b-kube-api-access-5vlq2\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-daemon-config\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-socket-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-node-log\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-cnibin\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-systemd-units\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.374153    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.374206    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:35.874188225 +0000 UTC m=+3.039515488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:35.375988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-kubelet-config\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-dbus\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-run-systemd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-kubelet\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbvj\" (UniqueName: \"kubernetes.io/projected/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-kube-api-access-gwbvj\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-tmp-dir\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-netd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-cni-netd\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-system-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ce78ece-1667-4ec6-be73-9f59da6e3b72-host-run-netns\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-socket-dir-parent\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-env-overrides\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-registration-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-system-cni-dir\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41323f0f-2da9-4852-bb4a-92ea2f770f87-host-slash\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.377446 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-script-lib\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-multus-socket-dir-parent\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cni-binary-copy\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41323f0f-2da9-4852-bb4a-92ea2f770f87-host-slash\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-multus-certs\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-registration-dir\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-host-run-multus-certs\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4h6\" (UniqueName: \"kubernetes.io/projected/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kube-api-access-ng4h6\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-etc-kubernetes\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.374809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0861aad9-818e-46fb-bef9-06e1d0d99c6b-etc-kubernetes\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.375238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84aa425b-634f-42cb-8758-a6980a14f99e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.375286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0861aad9-818e-46fb-bef9-06e1d0d99c6b-cni-binary-copy\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.375380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovnkube-script-lib\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.378230 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.375623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84aa425b-634f-42cb-8758-a6980a14f99e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.378790 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.378742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ce78ece-1667-4ec6-be73-9f59da6e3b72-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.386766 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.386698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9jf\" (UniqueName: \"kubernetes.io/projected/84aa425b-634f-42cb-8758-a6980a14f99e-kube-api-access-gk9jf\") pod \"multus-additional-cni-plugins-vfjds\" (UID: \"84aa425b-634f-42cb-8758-a6980a14f99e\") " pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.386901 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.386882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86wc\" (UniqueName: \"kubernetes.io/projected/41323f0f-2da9-4852-bb4a-92ea2f770f87-kube-api-access-r86wc\") pod \"iptables-alerter-g64bs\" (UID: \"41323f0f-2da9-4852-bb4a-92ea2f770f87\") " pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.387029 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.387007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97hq\" (UniqueName: \"kubernetes.io/projected/2ce78ece-1667-4ec6-be73-9f59da6e3b72-kube-api-access-d97hq\") pod \"ovnkube-node-dv7rn\" (UID: \"2ce78ece-1667-4ec6-be73-9f59da6e3b72\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.389509 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.389486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlq2\" (UniqueName: \"kubernetes.io/projected/0861aad9-818e-46fb-bef9-06e1d0d99c6b-kube-api-access-5vlq2\") pod \"multus-v2qvz\" (UID: \"0861aad9-818e-46fb-bef9-06e1d0d99c6b\") " pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.390934 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.390912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4h6\" (UniqueName: \"kubernetes.io/projected/ddbd7c07-791d-45fa-8d0d-0cedeb1116b9-kube-api-access-ng4h6\") pod \"aws-ebs-csi-driver-node-m89j4\" (UID: \"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.391030 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.390934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbvj\" (UniqueName: \"kubernetes.io/projected/56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8-kube-api-access-gwbvj\") pod \"node-resolver-qdng8\" (UID: \"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8\") " pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.468462 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.468425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s886d"
Sep 09 20:52:35.476116 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.476091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c7cjx"
Sep 09 20:52:35.485949 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.485925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:35.492935 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.492791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfjds"
Sep 09 20:52:35.501656 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.501632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qdng8"
Sep 09 20:52:35.508666 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.508644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:52:35.516651 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.516628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4"
Sep 09 20:52:35.525333 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.525314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2qvz"
Sep 09 20:52:35.533231 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.533210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g64bs"
Sep 09 20:52:35.778392 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.778307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:35.778549 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.778468    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:35.778549 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.778529    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.778509837 +0000 UTC m=+3.943837091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:35.879600 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.879562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:35.879800 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:35.879616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:35.879800 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879740    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:35.879800 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879770    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:35.879962 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879813    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.879772951 +0000 UTC m=+4.045100207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:35.879962 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879814    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:35.879962 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879830    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:35.879962 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:35.879893    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.879873573 +0000 UTC m=+4.045200839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:36.179934 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.179857    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:36.216459 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.216389    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:35 +0000 UTC" deadline="2028-03-15 09:17:46.65554018 +0000 UTC"
Sep 09 20:52:36.216459 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.216450    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13260h25m10.439094753s"
Sep 09 20:52:36.357262 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.357052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbd7c07_791d_45fa_8d0d_0cedeb1116b9.slice/crio-e47f013986cd5267c406942e874d7ccff2f7676035382df30958b214debb56ea WatchSource:0}: Error finding container e47f013986cd5267c406942e874d7ccff2f7676035382df30958b214debb56ea: Status 404 returned error can't find the container with id e47f013986cd5267c406942e874d7ccff2f7676035382df30958b214debb56ea
Sep 09 20:52:36.358363 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.358334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0861aad9_818e_46fb_bef9_06e1d0d99c6b.slice/crio-4b11a87d08ed7df0ab02c801048aac30d145230137a73e016d7208539ea8341a WatchSource:0}: Error finding container 4b11a87d08ed7df0ab02c801048aac30d145230137a73e016d7208539ea8341a: Status 404 returned error can't find the container with id 4b11a87d08ed7df0ab02c801048aac30d145230137a73e016d7208539ea8341a
Sep 09 20:52:36.359025 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.358993    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02664476_02ce_4f9e_9309_b18a425e76c6.slice/crio-9cf2608cd70f72c483e2fa74edc81efe473116fc692e975a82774f459e722b0b WatchSource:0}: Error finding container 9cf2608cd70f72c483e2fa74edc81efe473116fc692e975a82774f459e722b0b: Status 404 returned error can't find the container with id 9cf2608cd70f72c483e2fa74edc81efe473116fc692e975a82774f459e722b0b
Sep 09 20:52:36.360280 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.360254    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1219fa_b16b_4c7b_96f3_dea0ed0d3381.slice/crio-e4d156dbc2cfd8b7f98f0d7f014ca80ee1cd90bd0041f6678060aad7cb2c651d WatchSource:0}: Error finding container e4d156dbc2cfd8b7f98f0d7f014ca80ee1cd90bd0041f6678060aad7cb2c651d: Status 404 returned error can't find the container with id e4d156dbc2cfd8b7f98f0d7f014ca80ee1cd90bd0041f6678060aad7cb2c651d
Sep 09 20:52:36.361138 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.360808    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa425b_634f_42cb_8758_a6980a14f99e.slice/crio-1fb64c231e185a33a9de89ac2322a8970e82a750e99ff404b5731dde3433be09 WatchSource:0}: Error finding container 1fb64c231e185a33a9de89ac2322a8970e82a750e99ff404b5731dde3433be09: Status 404 returned error can't find the container with id 1fb64c231e185a33a9de89ac2322a8970e82a750e99ff404b5731dde3433be09
Sep 09 20:52:36.362174 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.361746    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c247cc_2776_45c8_9fae_f0374394b7a7.slice/crio-9ccaf985d759e90fbe61b6da73507a4e65e79bbfd581fbb37386c40bd1b9cd61 WatchSource:0}: Error finding container 9ccaf985d759e90fbe61b6da73507a4e65e79bbfd581fbb37386c40bd1b9cd61: Status 404 returned error can't find the container with id 9ccaf985d759e90fbe61b6da73507a4e65e79bbfd581fbb37386c40bd1b9cd61
Sep 09 20:52:36.363393 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.363303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d46fb7_8c1e_42b8_9dd8_42cf2b2f70d8.slice/crio-201389a9ce6d1817e8e33ea8035ea32af6c4cf727ad2f3845228ce93dbad9f4b WatchSource:0}: Error finding container 201389a9ce6d1817e8e33ea8035ea32af6c4cf727ad2f3845228ce93dbad9f4b: Status 404 returned error can't find the container with id 201389a9ce6d1817e8e33ea8035ea32af6c4cf727ad2f3845228ce93dbad9f4b
Sep 09 20:52:36.364058 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:52:36.363952    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41323f0f_2da9_4852_bb4a_92ea2f770f87.slice/crio-2f44a907ed6e839700bc05e79352614fcd0b9ecb4d3ae6e6c5046d7613ff2306 WatchSource:0}: Error finding container 2f44a907ed6e839700bc05e79352614fcd0b9ecb4d3ae6e6c5046d7613ff2306: Status 404 returned error can't find the container with id 2f44a907ed6e839700bc05e79352614fcd0b9ecb4d3ae6e6c5046d7613ff2306
Sep 09 20:52:36.786497 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.786391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:36.786665 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.786589    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:36.786734 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.786687    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:38.78666649 +0000 UTC m=+5.951993739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:36.887188 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.887143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:36.887188 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:36.887188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:36.887396 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887279    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:36.887396 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887331    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:38.887317491 +0000 UTC m=+6.052644738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:36.887509 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887411    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:36.887509 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887422    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:36.887509 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887432    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:36.887509 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:36.887458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:38.887448591 +0000 UTC m=+6.052775838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:37.217087 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.217026    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:35 +0000 UTC" deadline="2028-02-09 09:28:23.716367711 +0000 UTC"
Sep 09 20:52:37.217087 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.217064    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12420h35m46.499308495s"
Sep 09 20:52:37.278728 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.277911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:37.278728 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:37.278048    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:37.278728 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.278477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:37.278728 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:37.278575    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:37.278728 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.278649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:37.283939 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:37.279590    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:37.294589 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.294556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"87f638a04577e78afe73544e272688cb008137000cd5cadf4424c3e63ceceb73"}
Sep 09 20:52:37.301071 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.301041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g64bs" event={"ID":"41323f0f-2da9-4852-bb4a-92ea2f770f87","Type":"ContainerStarted","Data":"2f44a907ed6e839700bc05e79352614fcd0b9ecb4d3ae6e6c5046d7613ff2306"}
Sep 09 20:52:37.309862 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.309790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qdng8" event={"ID":"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8","Type":"ContainerStarted","Data":"201389a9ce6d1817e8e33ea8035ea32af6c4cf727ad2f3845228ce93dbad9f4b"}
Sep 09 20:52:37.317713 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.317682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s886d" event={"ID":"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381","Type":"ContainerStarted","Data":"e4d156dbc2cfd8b7f98f0d7f014ca80ee1cd90bd0041f6678060aad7cb2c651d"}
Sep 09 20:52:37.329025 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.328990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal" event={"ID":"3a96d69af6916da553cbc5c637b97bef","Type":"ContainerStarted","Data":"39d61ac44f5dd5e169a541822893b225a79283cabecd6847b2b0e48139295a5c"}
Sep 09 20:52:37.337532 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.337498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c7cjx" event={"ID":"02664476-02ce-4f9e-9309-b18a425e76c6","Type":"ContainerStarted","Data":"9cf2608cd70f72c483e2fa74edc81efe473116fc692e975a82774f459e722b0b"}
Sep 09 20:52:37.343320 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.343294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ldv7d" event={"ID":"e1c247cc-2776-45c8-9fae-f0374394b7a7","Type":"ContainerStarted","Data":"9ccaf985d759e90fbe61b6da73507a4e65e79bbfd581fbb37386c40bd1b9cd61"}
Sep 09 20:52:37.356130 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.356058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerStarted","Data":"1fb64c231e185a33a9de89ac2322a8970e82a750e99ff404b5731dde3433be09"}
Sep 09 20:52:37.371813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.371747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2qvz" event={"ID":"0861aad9-818e-46fb-bef9-06e1d0d99c6b","Type":"ContainerStarted","Data":"4b11a87d08ed7df0ab02c801048aac30d145230137a73e016d7208539ea8341a"}
Sep 09 20:52:37.378079 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.377955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4" event={"ID":"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9","Type":"ContainerStarted","Data":"e47f013986cd5267c406942e874d7ccff2f7676035382df30958b214debb56ea"}
Sep 09 20:52:37.392329 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.391374    2578 generic.go:356] "Generic (PLEG): container finished" podID="17e73fc6365d65c00c868dbb423c770e" containerID="39c4e5583b5a4d48f160ff6c4a01a26b3298e25a862357e42f67116393e832be" exitCode=0
Sep 09 20:52:37.392329 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.391413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal" event={"ID":"17e73fc6365d65c00c868dbb423c770e","Type":"ContainerDied","Data":"39c4e5583b5a4d48f160ff6c4a01a26b3298e25a862357e42f67116393e832be"}
Sep 09 20:52:37.414986 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:37.414332    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-161.ec2.internal" podStartSLOduration=2.414309483 podStartE2EDuration="2.414309483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:37.346119007 +0000 UTC m=+4.511446280" watchObservedRunningTime="2026-09-09 20:52:37.414309483 +0000 UTC m=+4.579636756"
Sep 09 20:52:38.411939 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:38.411895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal" event={"ID":"17e73fc6365d65c00c868dbb423c770e","Type":"ContainerStarted","Data":"5ff4b9de776ecd45fd4668b779250f489bf04e88793c0dbc382f7e3882e8ca55"}
Sep 09 20:52:38.431186 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:38.431123    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-161.ec2.internal" podStartSLOduration=4.43109746 podStartE2EDuration="4.43109746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:38.430423236 +0000 UTC m=+5.595750510" watchObservedRunningTime="2026-09-09 20:52:38.43109746 +0000 UTC m=+5.596424734"
Sep 09 20:52:38.805655 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:38.804888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:38.805655 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.805107    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:38.805655 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.805169    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:42.805151315 +0000 UTC m=+9.970478578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:38.906205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:38.906298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906383    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906401    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906416    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906428    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906472    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:42.90645067 +0000 UTC m=+10.071777945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:38.906490 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:38.906492    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:42.90648243 +0000 UTC m=+10.071809682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:39.278105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:39.277900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:39.278105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:39.277922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:39.278105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:39.278028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:39.278105 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:39.278039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:39.278458 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:39.278161    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:39.278458 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:39.278265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:41.279354 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:41.279319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:41.279831 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:41.279443    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:41.279903 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:41.279851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:41.279903 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:41.279865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:41.280005 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:41.279940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:41.280062 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:41.280038    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:42.841138 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:42.841104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:42.841575 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.841238    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:42.841575 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.841305    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.841286978 +0000 UTC m=+18.006614239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:42.942053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:42.942112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.942240    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.942300    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.942282389 +0000 UTC m=+18.107609648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.942679    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.942698    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:42.942793 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.942713    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:42.943301 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:42.943282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.942758539 +0000 UTC m=+18.108085800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:43.278156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:43.278161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:43.278260    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:43.278302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:43.278391    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:43.278574 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:43.278476    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:45.278604 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:45.278528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:45.278992 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:45.278646    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:45.279047 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:45.279018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:45.279120 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:45.279101    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:45.279166 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:45.279154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:45.279248 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:45.279230    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:47.277199 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:47.277165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:47.277545 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:47.277272    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:47.277545 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:47.277174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:47.277545 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:47.277311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:47.277545 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:47.277361    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:47.277545 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:47.277409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:49.278051 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:49.277815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:49.278482 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:49.277829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:49.278482 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:49.278142    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:49.278482 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:49.277890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:49.278482 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:49.278235    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:49.278482 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:49.278316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:50.902726 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:50.902687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:50.903221 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:50.902860    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:50.903221 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:50.902934    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.902914546 +0000 UTC m=+34.068241807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:51.003505 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:51.003473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:51.003505 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:51.003511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003626    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003634    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003657    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003670    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.003668839 +0000 UTC m=+34.168996084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:51.003716 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.003713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.003701153 +0000 UTC m=+34.169028399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:51.277657 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:51.277592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:51.277802 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:51.277600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:51.277802 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.277696    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:51.277880 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.277803    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:51.277880 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:51.277600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:51.277940 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:51.277882    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:53.277856 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:53.277823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:53.278312 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:53.277926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:53.278312 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:53.278015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:53.278312 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:53.278126    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:53.278312 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:53.278188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:53.278312 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:53.278270    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:55.277362 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.277198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:55.277877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.277220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:55.277877 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:55.277455    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:55.277877 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:55.277513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:55.277877 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.277268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:55.277877 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:55.277610    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:55.446914 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.446696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2qvz" event={"ID":"0861aad9-818e-46fb-bef9-06e1d0d99c6b","Type":"ContainerStarted","Data":"176aa40e115f82c64846fd5e638af6f41eb25e907aa168c6fac1c93de4cd0b40"}
Sep 09 20:52:55.448555 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.448526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4" event={"ID":"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9","Type":"ContainerStarted","Data":"9805a575f22dfd7efa9cc70fb8545a28d4170f53a3b6efbbcda2896d3d5f1ad1"}
Sep 09 20:52:55.452208 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"8197dd783db613950d845789c328d4f3f9c8fb198f2d1628701ed240c74809be"}
Sep 09 20:52:55.452309 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"6db966ed38a42a3ad06a804edb2a2b63ef314e218e1b1bdedd66d916515c5ca4"}
Sep 09 20:52:55.452309 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"045f28261937e980772c914aa86a1bf896c825d9950d149aa84384129ef4336a"}
Sep 09 20:52:55.452382 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"8c94acf1d6dc09c85249e687443e297d7fe4119154d683fd2dabaf07094088e0"}
Sep 09 20:52:55.452382 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"3e9f79f1a83fc8240c561f599d7f7a3cee1884092cb8338218167108b4004c06"}
Sep 09 20:52:55.452382 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.452380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"2971ff9f921fdcd0d840af51e6eff998d58b9d86f2c9324aaf620d3bbd7c05dd"}
Sep 09 20:52:55.453863 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.453836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qdng8" event={"ID":"56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8","Type":"ContainerStarted","Data":"3fbd4e4d44ab8937b806cf95db10deeb4903eb9c1b6c54e7c430a1a995949640"}
Sep 09 20:52:55.455949 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.455922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s886d" event={"ID":"cb1219fa-b16b-4c7b-96f3-dea0ed0d3381","Type":"ContainerStarted","Data":"8223f50add08ff0183978757318cd385708c5304a2dccbc549671e442203a313"}
Sep 09 20:52:55.457732 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.457710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c7cjx" event={"ID":"02664476-02ce-4f9e-9309-b18a425e76c6","Type":"ContainerStarted","Data":"444e5514630f11a394a95da8df6624a1ea106224f8af92d388f440affc3e52b1"}
Sep 09 20:52:55.459571 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.459547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ldv7d" event={"ID":"e1c247cc-2776-45c8-9fae-f0374394b7a7","Type":"ContainerStarted","Data":"f4822ea06d29b83c15257c0e84c6a1986bcdd3835639d3af1b8c841463839e6e"}
Sep 09 20:52:55.462390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.461694    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="b31324aea36514b17379f7ff6c528fdf42c05d463fbaececab6b525906c601a9" exitCode=0
Sep 09 20:52:55.462390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.461728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"b31324aea36514b17379f7ff6c528fdf42c05d463fbaececab6b525906c601a9"}
Sep 09 20:52:55.484390 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.484340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v2qvz" podStartSLOduration=4.03085864 podStartE2EDuration="22.484321551s" totalImagesPullingTime="18.453462911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.465470433 +0000 UTC m=+22.630797728" watchObservedRunningTime="2026-09-09 20:52:55.484321551 +0000 UTC m=+22.649648823"
Sep 09 20:52:55.484804 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.484780    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c7cjx" podStartSLOduration=4.095453343 podStartE2EDuration="22.484771084s" totalImagesPullingTime="18.389317741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.484486129 +0000 UTC m=+22.649813397" watchObservedRunningTime="2026-09-09 20:52:55.484771084 +0000 UTC m=+22.650098354"
Sep 09 20:52:55.502760 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.502671    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ldv7d" podStartSLOduration=11.472988117 podStartE2EDuration="22.502660378s" totalImagesPullingTime="11.029672261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.502496411 +0000 UTC m=+22.667823678" watchObservedRunningTime="2026-09-09 20:52:55.502660378 +0000 UTC m=+22.667987645"
Sep 09 20:52:55.520188 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:55.520155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s886d" podStartSLOduration=4.113607989 podStartE2EDuration="22.520145103s" totalImagesPullingTime="18.406537114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.520013155 +0000 UTC m=+22.685340423" watchObservedRunningTime="2026-09-09 20:52:55.520145103 +0000 UTC m=+22.685472370"
Sep 09 20:52:56.209042 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.209018    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:52:56.226724 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.226647    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:56.209039514Z","UUID":"041ce6e0-e076-4d72-bec9-4fb1abad63b8","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:52:56.227899 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.227885    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:52:56.227965 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.227912    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:52:56.467684 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.467609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4" event={"ID":"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9","Type":"ContainerStarted","Data":"e4e649e93c6af70f585b5ef3b76af04a97ebe10c3acbfd76a050e67a99b69db5"}
Sep 09 20:52:56.469243 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.469215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g64bs" event={"ID":"41323f0f-2da9-4852-bb4a-92ea2f770f87","Type":"ContainerStarted","Data":"da13c52579e5aabf28da38c4c27f3e79e3523899a8b28c24f6aa165acb2acb36"}
Sep 09 20:52:56.492618 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:56.492572    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qdng8" podStartSLOduration=5.10741165 podStartE2EDuration="23.492557781s" totalImagesPullingTime="18.385146131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:55.56202017 +0000 UTC m=+22.727347441" watchObservedRunningTime="2026-09-09 20:52:56.492557781 +0000 UTC m=+23.657885049"
Sep 09 20:52:57.276896 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:57.276863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:57.277056 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:57.276908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:57.277056 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:57.276863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:57.277056 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:57.277016    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:57.277219 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:57.277147    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:57.277273 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:57.277240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:58.477182 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.476957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4" event={"ID":"ddbd7c07-791d-45fa-8d0d-0cedeb1116b9","Type":"ContainerStarted","Data":"42609d4da21b0f36a8e33c2e9eccbbb686a2868b2c486b849dbac843400e89b2"}
Sep 09 20:52:58.480942 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.480913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"5d1c5995849ee0db6643002d6714b472454f840da52ec9a50f8fc589aaac6151"}
Sep 09 20:52:58.499716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.499663    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g64bs" podStartSLOduration=7.116765795 podStartE2EDuration="25.499646666s" totalImagesPullingTime="18.382880871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:56.492954283 +0000 UTC m=+23.658281578" watchObservedRunningTime="2026-09-09 20:52:58.499646666 +0000 UTC m=+25.664974107"
Sep 09 20:52:58.500025 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.499986    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m89j4" podStartSLOduration=4.369534208 podStartE2EDuration="25.499976503s" totalImagesPullingTime="21.130442295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:58.498478302 +0000 UTC m=+25.663805573" watchObservedRunningTime="2026-09-09 20:52:58.499976503 +0000 UTC m=+25.665303771"
Sep 09 20:52:58.654913 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.654888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:58.655531 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:58.655515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:59.277542 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:59.277510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:52:59.277542 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:59.277527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:52:59.277845 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:59.277548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:52:59.277845 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:59.277637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:52:59.277915 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:59.277875    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:52:59.278026 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:52:59.278000    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:52:59.483642 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:59.483603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:52:59.484207 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:52:59.484184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ldv7d"
Sep 09 20:53:00.487808 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.487632    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="e3e8ad49f2ed135708372918e79cac71fa061590c02db58f2f2281b92eba24ba" exitCode=0
Sep 09 20:53:00.488185 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.487713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"e3e8ad49f2ed135708372918e79cac71fa061590c02db58f2f2281b92eba24ba"}
Sep 09 20:53:00.491603 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.491579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" event={"ID":"2ce78ece-1667-4ec6-be73-9f59da6e3b72","Type":"ContainerStarted","Data":"279089158d95dbdf3e922d5939535fb2c7ff4c53663efcf9b1571fe28e44200d"}
Sep 09 20:53:00.491927 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.491904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:00.492045 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.492032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:00.492105 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.492089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:00.507219 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.507199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:00.507300 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.507290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:00.541400 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:00.541360    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn" podStartSLOduration=9.096426507 podStartE2EDuration="27.541348336s" totalImagesPullingTime="18.444921829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.539908314 +0000 UTC m=+27.705235582" watchObservedRunningTime="2026-09-09 20:53:00.541348336 +0000 UTC m=+27.706675604"
Sep 09 20:53:01.277210 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.277072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:01.277210 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.277095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:01.277210 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.277072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:01.277210 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.277176    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:53:01.277581 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.277259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:53:01.277581 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.277317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:53:01.895804 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.895692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5k56s"]
Sep 09 20:53:01.896786 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.895844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:01.896786 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.895935    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:53:01.897198 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.897173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wzrr"]
Sep 09 20:53:01.897309 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.897271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:01.897375 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.897352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:53:01.914229 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.914207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9g5pw"]
Sep 09 20:53:01.914336 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:01.914300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:01.914404 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:01.914372    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:53:02.497660 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:02.497623    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="0048d0c180c05301a02a5923f711b820b731e047851ea35fe82fa00dca2747e8" exitCode=0
Sep 09 20:53:02.497846 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:02.497702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"0048d0c180c05301a02a5923f711b820b731e047851ea35fe82fa00dca2747e8"}
Sep 09 20:53:03.278387 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:03.278226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:03.278713 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:03.278291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:03.278713 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:03.278453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:53:03.278713 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:03.278309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:03.278713 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:03.278536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:53:03.278713 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:03.278626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:53:04.508591 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:04.508558    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="aea8fb88926e88c390001d292f9d98275810ceaa03a7b4df7f0d11e01020adea" exitCode=0
Sep 09 20:53:04.509079 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:04.508630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"aea8fb88926e88c390001d292f9d98275810ceaa03a7b4df7f0d11e01020adea"}
Sep 09 20:53:05.277662 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:05.277633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:05.277856 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:05.277730    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:53:05.277856 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:05.277635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:05.277856 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:05.277778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:05.277856 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:05.277849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:53:05.278009 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:05.277903    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:53:06.925622 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:06.925547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:06.926050 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:06.925688    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:06.926050 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:06.925769    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs podName:ee1b4883-3bbb-459e-b700-5a38c2131ba3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:38.925733365 +0000 UTC m=+66.091060625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs") pod "network-metrics-daemon-5k56s" (UID: "ee1b4883-3bbb-459e-b700-5a38c2131ba3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:07.026444 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:07.026413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:07.026605 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:07.026463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:07.026605 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026589    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:07.026712 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026615    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:07.026712 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026615    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:07.026712 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026628    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9ckvd for pod openshift-network-diagnostics/network-check-target-9g5pw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:07.026712 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret podName:2d89900e-61d0-4ad4-9e29-8ffdc512f12b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:39.026663813 +0000 UTC m=+66.191991059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret") pod "global-pull-secret-syncer-4wzrr" (UID: "2d89900e-61d0-4ad4-9e29-8ffdc512f12b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:07.026712 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.026693    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd podName:14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:39.026687936 +0000 UTC m=+66.192015183 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ckvd" (UniqueName: "kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd") pod "network-check-target-9g5pw" (UID: "14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:07.276827 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:07.276756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:07.276827 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:07.276794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:07.277035 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:07.276741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:07.277035 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.276884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k56s" podUID="ee1b4883-3bbb-459e-b700-5a38c2131ba3"
Sep 09 20:53:07.277035 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.276955    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wzrr" podUID="2d89900e-61d0-4ad4-9e29-8ffdc512f12b"
Sep 09 20:53:07.277035 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:07.277010    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g5pw" podUID="14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5"
Sep 09 20:53:08.336089 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.336058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qdng8_56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8/dns-node-resolver/0.log"
Sep 09 20:53:08.669275 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.669202    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-161.ec2.internal" event="NodeReady"
Sep 09 20:53:08.669428 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.669334    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:53:08.730393 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.730358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kw5d2"]
Sep 09 20:53:08.732905 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.732885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2qt4t"]
Sep 09 20:53:08.733022 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.733000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.734415 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.734378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.735606 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.735587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:53:08.735691 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.735609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:53:08.737192 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.737155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:53:08.737192 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.737165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rcsnd\""
Sep 09 20:53:08.737192 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.737157    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:53:08.737528 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.737508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:53:08.738214 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.738196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6lpmt\""
Sep 09 20:53:08.750649 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.750172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qt4t"]
Sep 09 20:53:08.750649 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.750467    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5d2"]
Sep 09 20:53:08.763668 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.763649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gk2zv"]
Sep 09 20:53:08.765441 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.765424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.769257 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.769236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:53:08.769368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.769266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:53:08.769368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.769271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:53:08.769368 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.769270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:53:08.769987 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.769970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fptv8\""
Sep 09 20:53:08.784152 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.784135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk2zv"]
Sep 09 20:53:08.839457 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.839546 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mw4x\" (UniqueName: \"kubernetes.io/projected/d617a952-4648-4c17-9447-69f97fbbe210-kube-api-access-6mw4x\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.839546 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69efb2bd-e552-4e15-8b96-d79aad2358a5-cert\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.839546 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.839712 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.839712 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zvm\" (UniqueName: \"kubernetes.io/projected/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-api-access-c5zvm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.839834 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlwd\" (UniqueName: \"kubernetes.io/projected/69efb2bd-e552-4e15-8b96-d79aad2358a5-kube-api-access-2rlwd\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.839834 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d617a952-4648-4c17-9447-69f97fbbe210-config-volume\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.839834 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d617a952-4648-4c17-9447-69f97fbbe210-tmp-dir\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.839962 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3bf0307-f669-46a2-b79f-a7038ffe5b74-crio-socket\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.839962 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.839870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3bf0307-f669-46a2-b79f-a7038ffe5b74-data-volume\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.940592 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.940592 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zvm\" (UniqueName: \"kubernetes.io/projected/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-api-access-c5zvm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.940761 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:08.940695    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:08.940802 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlwd\" (UniqueName: \"kubernetes.io/projected/69efb2bd-e552-4e15-8b96-d79aad2358a5-kube-api-access-2rlwd\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.940802 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:08.940785    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls podName:e3bf0307-f669-46a2-b79f-a7038ffe5b74 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.440763547 +0000 UTC m=+36.606090807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk2zv" (UID: "e3bf0307-f669-46a2-b79f-a7038ffe5b74") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:08.940894 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d617a952-4648-4c17-9447-69f97fbbe210-config-volume\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.940894 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d617a952-4648-4c17-9447-69f97fbbe210-tmp-dir\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.940988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3bf0307-f669-46a2-b79f-a7038ffe5b74-crio-socket\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.940988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3bf0307-f669-46a2-b79f-a7038ffe5b74-data-volume\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.940988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.941125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.940995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mw4x\" (UniqueName: \"kubernetes.io/projected/d617a952-4648-4c17-9447-69f97fbbe210-kube-api-access-6mw4x\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.941125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69efb2bd-e552-4e15-8b96-d79aad2358a5-cert\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.941125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.941125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3bf0307-f669-46a2-b79f-a7038ffe5b74-crio-socket\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.941303 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:08.941154    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:08.941303 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:08.941198    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls podName:d617a952-4648-4c17-9447-69f97fbbe210 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.44118312 +0000 UTC m=+36.606510389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls") pod "dns-default-kw5d2" (UID: "d617a952-4648-4c17-9447-69f97fbbe210") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:08.941303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d617a952-4648-4c17-9447-69f97fbbe210-tmp-dir\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.941303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3bf0307-f669-46a2-b79f-a7038ffe5b74-data-volume\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.941469 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d617a952-4648-4c17-9447-69f97fbbe210-config-volume\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.941588 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.941572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:08.946112 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.945945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69efb2bd-e552-4e15-8b96-d79aad2358a5-cert\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.959904 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.959875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlwd\" (UniqueName: \"kubernetes.io/projected/69efb2bd-e552-4e15-8b96-d79aad2358a5-kube-api-access-2rlwd\") pod \"ingress-canary-2qt4t\" (UID: \"69efb2bd-e552-4e15-8b96-d79aad2358a5\") " pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:08.960113 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.960096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mw4x\" (UniqueName: \"kubernetes.io/projected/d617a952-4648-4c17-9447-69f97fbbe210-kube-api-access-6mw4x\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:08.960292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:08.960269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zvm\" (UniqueName: \"kubernetes.io/projected/e3bf0307-f669-46a2-b79f-a7038ffe5b74-kube-api-access-c5zvm\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:09.054452 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.054423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qt4t"
Sep 09 20:53:09.062366 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.062344    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-g658j"]
Sep 09 20:53:09.064240 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.064225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.068078 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.068056    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 20:53:09.068182 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.068088    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 20:53:09.068182 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.068140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 20:53:09.068797 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.068778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j47kc\""
Sep 09 20:53:09.068888 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.068801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 20:53:09.078064 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.078045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-g658j"]
Sep 09 20:53:09.118043 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.118024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c7cjx_02664476-02ce-4f9e-9309-b18a425e76c6/node-ca/0.log"
Sep 09 20:53:09.142494 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.142453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjwx\" (UniqueName: \"kubernetes.io/projected/170d9ba2-dd10-4e0c-9205-336353db1479-kube-api-access-lxjwx\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.142592 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.142503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/170d9ba2-dd10-4e0c-9205-336353db1479-signing-cabundle\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.142592 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.142546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/170d9ba2-dd10-4e0c-9205-336353db1479-signing-key\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.243426 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.243374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/170d9ba2-dd10-4e0c-9205-336353db1479-signing-key\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.243508 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.243434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjwx\" (UniqueName: \"kubernetes.io/projected/170d9ba2-dd10-4e0c-9205-336353db1479-kube-api-access-lxjwx\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.243508 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.243458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/170d9ba2-dd10-4e0c-9205-336353db1479-signing-cabundle\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.244042 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.244013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/170d9ba2-dd10-4e0c-9205-336353db1479-signing-cabundle\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.245967 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.245949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/170d9ba2-dd10-4e0c-9205-336353db1479-signing-key\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.253972 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.253954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjwx\" (UniqueName: \"kubernetes.io/projected/170d9ba2-dd10-4e0c-9205-336353db1479-kube-api-access-lxjwx\") pod \"service-ca-75d99c484-g658j\" (UID: \"170d9ba2-dd10-4e0c-9205-336353db1479\") " pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.277038 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.277016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:09.277125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.277018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:09.277267 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.277248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:09.282339 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.282324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:09.283511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.283254    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:09.283511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.283313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pstzn\""
Sep 09 20:53:09.283511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.283329    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:09.283511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.283403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:09.283831 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.283648    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-thpvv\""
Sep 09 20:53:09.341906 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.341888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qt4t"]
Sep 09 20:53:09.343898 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:09.343877    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69efb2bd_e552_4e15_8b96_d79aad2358a5.slice/crio-f3fa40867018a63799399bc57172c863cfeaa1b28a8c7cd1ae8644682bc9fd86 WatchSource:0}: Error finding container f3fa40867018a63799399bc57172c863cfeaa1b28a8c7cd1ae8644682bc9fd86: Status 404 returned error can't find the container with id f3fa40867018a63799399bc57172c863cfeaa1b28a8c7cd1ae8644682bc9fd86
Sep 09 20:53:09.377343 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.377325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-g658j"
Sep 09 20:53:09.444111 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.444088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:09.444226 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.444207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:09.444277 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:09.444240    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:09.444311 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:09.444300    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls podName:e3bf0307-f669-46a2-b79f-a7038ffe5b74 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.444280744 +0000 UTC m=+37.609608013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk2zv" (UID: "e3bf0307-f669-46a2-b79f-a7038ffe5b74") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:09.444311 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:09.444300    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:09.444372 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:09.444334    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls podName:d617a952-4648-4c17-9447-69f97fbbe210 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.444323997 +0000 UTC m=+37.609651260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls") pod "dns-default-kw5d2" (UID: "d617a952-4648-4c17-9447-69f97fbbe210") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:09.524197 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.524138    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="535f51fa2da78940747cd6e5204a31208ba224cdaea06f995591c48a6c9147a9" exitCode=0
Sep 09 20:53:09.524287 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.524213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"535f51fa2da78940747cd6e5204a31208ba224cdaea06f995591c48a6c9147a9"}
Sep 09 20:53:09.525427 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.525403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qt4t" event={"ID":"69efb2bd-e552-4e15-8b96-d79aad2358a5","Type":"ContainerStarted","Data":"f3fa40867018a63799399bc57172c863cfeaa1b28a8c7cd1ae8644682bc9fd86"}
Sep 09 20:53:09.648440 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:09.648406    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-g658j"]
Sep 09 20:53:09.676372 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:09.676352    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170d9ba2_dd10_4e0c_9205_336353db1479.slice/crio-97204dcdcfa2eb0ee3ab4614fd920cffda504118a1593d712a5f8564b373c83d WatchSource:0}: Error finding container 97204dcdcfa2eb0ee3ab4614fd920cffda504118a1593d712a5f8564b373c83d: Status 404 returned error can't find the container with id 97204dcdcfa2eb0ee3ab4614fd920cffda504118a1593d712a5f8564b373c83d
Sep 09 20:53:10.453922 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:10.453892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:10.454450 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:10.453937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:10.454450 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:10.454080    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:10.454450 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:10.454144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls podName:d617a952-4648-4c17-9447-69f97fbbe210 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:12.454128328 +0000 UTC m=+39.619455574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls") pod "dns-default-kw5d2" (UID: "d617a952-4648-4c17-9447-69f97fbbe210") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:10.454450 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:10.454085    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:10.454450 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:10.454224    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls podName:e3bf0307-f669-46a2-b79f-a7038ffe5b74 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:12.454212096 +0000 UTC m=+39.619539341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk2zv" (UID: "e3bf0307-f669-46a2-b79f-a7038ffe5b74") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:10.531336 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:10.531301    2578 generic.go:356] "Generic (PLEG): container finished" podID="84aa425b-634f-42cb-8758-a6980a14f99e" containerID="84cd432daa82156c483cf7306c8333b8bae96082b59fa0d809909456963462c2" exitCode=0
Sep 09 20:53:10.531491 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:10.531387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerDied","Data":"84cd432daa82156c483cf7306c8333b8bae96082b59fa0d809909456963462c2"}
Sep 09 20:53:10.532925 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:10.532902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-g658j" event={"ID":"170d9ba2-dd10-4e0c-9205-336353db1479","Type":"ContainerStarted","Data":"97204dcdcfa2eb0ee3ab4614fd920cffda504118a1593d712a5f8564b373c83d"}
Sep 09 20:53:11.542061 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:11.541717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfjds" event={"ID":"84aa425b-634f-42cb-8758-a6980a14f99e","Type":"ContainerStarted","Data":"2c252b6c5716bcaa9d4fb73802f2c763c37ef9476ead1759c1496e5c08aecd86"}
Sep 09 20:53:11.582828 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:11.582772    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vfjds" podStartSLOduration=9.308871123 podStartE2EDuration="38.58273545s" totalImagesPullingTime="29.273864327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:11.582426654 +0000 UTC m=+38.747753923" watchObservedRunningTime="2026-09-09 20:53:11.58273545 +0000 UTC m=+38.748062708"
Sep 09 20:53:12.472850 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:12.472813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:12.472850 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:12.472852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:12.473022 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:12.472953    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:12.473022 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:12.472952    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:12.473022 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:12.473005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls podName:e3bf0307-f669-46a2-b79f-a7038ffe5b74 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.472991463 +0000 UTC m=+43.638318708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk2zv" (UID: "e3bf0307-f669-46a2-b79f-a7038ffe5b74") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:12.473022 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:12.473018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls podName:d617a952-4648-4c17-9447-69f97fbbe210 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.473012065 +0000 UTC m=+43.638339310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls") pod "dns-default-kw5d2" (UID: "d617a952-4648-4c17-9447-69f97fbbe210") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:13.546884 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:13.546850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qt4t" event={"ID":"69efb2bd-e552-4e15-8b96-d79aad2358a5","Type":"ContainerStarted","Data":"d4a9828ce6d781e0ed8de82e00815ea89e3997ccb78d6d3ae74abf6e432ad16b"}
Sep 09 20:53:13.548304 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:13.548274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-g658j" event={"ID":"170d9ba2-dd10-4e0c-9205-336353db1479","Type":"ContainerStarted","Data":"e362feae5a67e559e6c0eba416017195c280f0ad625c2c994416b6aab8fe627b"}
Sep 09 20:53:13.566210 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:13.566166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2qt4t" podStartSLOduration=2.335377556 podStartE2EDuration="5.566152679s" totalImagesPullingTime="3.230775123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:13.56536551 +0000 UTC m=+40.730692772" watchObservedRunningTime="2026-09-09 20:53:13.566152679 +0000 UTC m=+40.731479946"
Sep 09 20:53:13.583884 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:13.583845    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-g658j" podStartSLOduration=1.679963492 podStartE2EDuration="4.583832094s" totalImagesPullingTime="2.903868602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:13.583399714 +0000 UTC m=+40.748726982" watchObservedRunningTime="2026-09-09 20:53:13.583832094 +0000 UTC m=+40.749159372"
Sep 09 20:53:16.503153 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:16.503115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:16.503153 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:16.503156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:16.503665 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:16.503263    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:16.503665 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:16.503331    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls podName:d617a952-4648-4c17-9447-69f97fbbe210 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:24.50331547 +0000 UTC m=+51.668642721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls") pod "dns-default-kw5d2" (UID: "d617a952-4648-4c17-9447-69f97fbbe210") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:16.503665 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:16.503267    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:16.503665 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:53:16.503397    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls podName:e3bf0307-f669-46a2-b79f-a7038ffe5b74 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:24.503385215 +0000 UTC m=+51.668712460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk2zv" (UID: "e3bf0307-f669-46a2-b79f-a7038ffe5b74") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:24.561551 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.561517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:24.561551 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.561557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:24.565190 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.565165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3bf0307-f669-46a2-b79f-a7038ffe5b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk2zv\" (UID: \"e3bf0307-f669-46a2-b79f-a7038ffe5b74\") " pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:24.565286 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.565165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d617a952-4648-4c17-9447-69f97fbbe210-metrics-tls\") pod \"dns-default-kw5d2\" (UID: \"d617a952-4648-4c17-9447-69f97fbbe210\") " pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:24.645241 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.645215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:24.673952 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.673931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk2zv"
Sep 09 20:53:24.943706 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.943656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5d2"]
Sep 09 20:53:24.945499 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:24.945476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd617a952_4648_4c17_9447_69f97fbbe210.slice/crio-2f3014c6ddbc0e8474e04523fc150f8a221ae44eefb8bd0a81f6ee5f23ae7597 WatchSource:0}: Error finding container 2f3014c6ddbc0e8474e04523fc150f8a221ae44eefb8bd0a81f6ee5f23ae7597: Status 404 returned error can't find the container with id 2f3014c6ddbc0e8474e04523fc150f8a221ae44eefb8bd0a81f6ee5f23ae7597
Sep 09 20:53:24.978959 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:24.978939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk2zv"]
Sep 09 20:53:24.981713 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:24.981687    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bf0307_f669_46a2_b79f_a7038ffe5b74.slice/crio-cc7dccb17c753b07b97773f6668b60b3c67670fa5d989e054cd9cb386fef1653 WatchSource:0}: Error finding container cc7dccb17c753b07b97773f6668b60b3c67670fa5d989e054cd9cb386fef1653: Status 404 returned error can't find the container with id cc7dccb17c753b07b97773f6668b60b3c67670fa5d989e054cd9cb386fef1653
Sep 09 20:53:25.577999 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:25.577971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk2zv" event={"ID":"e3bf0307-f669-46a2-b79f-a7038ffe5b74","Type":"ContainerStarted","Data":"db6d3b1cb3f8b2f7e21ce28470940c87e87fd5dc7d184ed627808e1100798d06"}
Sep 09 20:53:25.578384 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:25.578024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk2zv" event={"ID":"e3bf0307-f669-46a2-b79f-a7038ffe5b74","Type":"ContainerStarted","Data":"cc7dccb17c753b07b97773f6668b60b3c67670fa5d989e054cd9cb386fef1653"}
Sep 09 20:53:25.579025 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:25.578998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5d2" event={"ID":"d617a952-4648-4c17-9447-69f97fbbe210","Type":"ContainerStarted","Data":"2f3014c6ddbc0e8474e04523fc150f8a221ae44eefb8bd0a81f6ee5f23ae7597"}
Sep 09 20:53:26.583905 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:26.583874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk2zv" event={"ID":"e3bf0307-f669-46a2-b79f-a7038ffe5b74","Type":"ContainerStarted","Data":"0780538d7b86f0d9f29007c58cc8f0e57508f9ce864a2eeeccf3616dcf2ad780"}
Sep 09 20:53:27.589006 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:27.588972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5d2" event={"ID":"d617a952-4648-4c17-9447-69f97fbbe210","Type":"ContainerStarted","Data":"a006a2e75ea06c7fec5e6f23340cb0bbde4a22f2fe73acfa8a826a2532f8c8a2"}
Sep 09 20:53:27.589413 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:27.589022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5d2" event={"ID":"d617a952-4648-4c17-9447-69f97fbbe210","Type":"ContainerStarted","Data":"af32cd1de6f206173f7788394e3df8d7bdfb57cea44c9bb1abdf386a009fc6ce"}
Sep 09 20:53:27.589413 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:27.589109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:27.611152 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:27.611085    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kw5d2" podStartSLOduration=17.806659312 podStartE2EDuration="19.611067597s" totalImagesPullingTime="1.804408285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:27.60981222 +0000 UTC m=+54.775139489" watchObservedRunningTime="2026-09-09 20:53:27.611067597 +0000 UTC m=+54.776394864"
Sep 09 20:53:28.594469 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:28.594430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk2zv" event={"ID":"e3bf0307-f669-46a2-b79f-a7038ffe5b74","Type":"ContainerStarted","Data":"8f99b3f634cca61f86f0c2793f55dd9e6fd45a1b82f1321a9339a87da8470727"}
Sep 09 20:53:28.614973 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:28.614922    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gk2zv" podStartSLOduration=18.178792061 podStartE2EDuration="20.614906461s" totalImagesPullingTime="2.4361144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:28.614528718 +0000 UTC m=+55.779855997" watchObservedRunningTime="2026-09-09 20:53:28.614906461 +0000 UTC m=+55.780233729"
Sep 09 20:53:31.317232 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.317202    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"]
Sep 09 20:53:31.335699 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.335673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.360353 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.360328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:53:31.360494 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.360339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:53:31.364589 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.364569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:53:31.364876 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.364850    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:53:31.365003 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.364954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:53:31.365060 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.365042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:53:31.387623 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.387606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:53:31.406188 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.406292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.406292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.406292 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.406418 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.406418 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.406323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr9v\" (UniqueName: \"kubernetes.io/projected/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-kube-api-access-xdr9v\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.410661 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.410641    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"]
Sep 09 20:53:31.478323 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.478304    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-zgc8d"]
Sep 09 20:53:31.489602 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.489581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:31.496632 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.496615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:31.497786 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.497770    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:31.497903 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.497889    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m6wbd\""
Sep 09 20:53:31.507300 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.507388 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.507388 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.507504 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.507504 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp298\" (UniqueName: \"kubernetes.io/projected/9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8-kube-api-access-wp298\") pod \"downloads-85d98c998f-zgc8d\" (UID: \"9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8\") " pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:31.507504 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.507692 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.507499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr9v\" (UniqueName: \"kubernetes.io/projected/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-kube-api-access-xdr9v\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.508359 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.508336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.509945 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.509922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-ca\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.510015 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.509929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.510015 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.509982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.510082 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.510050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.516882 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.516862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-zgc8d"]
Sep 09 20:53:31.547779 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.547732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr9v\" (UniqueName: \"kubernetes.io/projected/b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4-kube-api-access-xdr9v\") pod \"cluster-proxy-proxy-agent-ddfcc5d57-kbrx9\" (UID: \"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.608706 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.608678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp298\" (UniqueName: \"kubernetes.io/projected/9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8-kube-api-access-wp298\") pod \"downloads-85d98c998f-zgc8d\" (UID: \"9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8\") " pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:31.631383 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.631364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp298\" (UniqueName: \"kubernetes.io/projected/9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8-kube-api-access-wp298\") pod \"downloads-85d98c998f-zgc8d\" (UID: \"9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8\") " pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:31.651276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.651249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"
Sep 09 20:53:31.798143 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.798116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:31.972895 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:31.972867    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9"]
Sep 09 20:53:31.977619 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:31.977590    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4954e1b_9cab_4f0d_a3b8_2051a7c3b5a4.slice/crio-7b1d90e6f3eb7e2638fd7da91f8bda90b10bcf73b91351e45c48e3875ad893f5 WatchSource:0}: Error finding container 7b1d90e6f3eb7e2638fd7da91f8bda90b10bcf73b91351e45c48e3875ad893f5: Status 404 returned error can't find the container with id 7b1d90e6f3eb7e2638fd7da91f8bda90b10bcf73b91351e45c48e3875ad893f5
Sep 09 20:53:32.097822 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:32.097800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-zgc8d"]
Sep 09 20:53:32.100334 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:32.100313    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eeccdf2_6b0f_4c60_96b1_dbdb97e819d8.slice/crio-f527778e8e35d33d124965dabb492802863a80bd17316370b7f71a9c96f29dd7 WatchSource:0}: Error finding container f527778e8e35d33d124965dabb492802863a80bd17316370b7f71a9c96f29dd7: Status 404 returned error can't find the container with id f527778e8e35d33d124965dabb492802863a80bd17316370b7f71a9c96f29dd7
Sep 09 20:53:32.517673 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:32.517648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7rn"
Sep 09 20:53:32.611176 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:32.611146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9" event={"ID":"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4","Type":"ContainerStarted","Data":"7b1d90e6f3eb7e2638fd7da91f8bda90b10bcf73b91351e45c48e3875ad893f5"}
Sep 09 20:53:32.612120 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:32.612099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-zgc8d" event={"ID":"9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8","Type":"ContainerStarted","Data":"f527778e8e35d33d124965dabb492802863a80bd17316370b7f71a9c96f29dd7"}
Sep 09 20:53:36.631531 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:36.631492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9" event={"ID":"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4","Type":"ContainerStarted","Data":"23c3a2ee7c11a1dd2bce52d2097178bb389a642a76e9de402eb27f663241803d"}
Sep 09 20:53:36.631998 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:36.631541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9" event={"ID":"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4","Type":"ContainerStarted","Data":"0d23184cd07ed8a5be6ad2e63b8f5e2252d94d164be787e72b21bea15243ba95"}
Sep 09 20:53:36.631998 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:36.631558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9" event={"ID":"b4954e1b-9cab-4f0d-a3b8-2051a7c3b5a4","Type":"ContainerStarted","Data":"1ec16bb4ae1fa79505ae5b6bf2abfc489b71b9a1a147496b2844c4e5b59fc022"}
Sep 09 20:53:36.656415 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:36.656370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ddfcc5d57-kbrx9" podStartSLOduration=1.501507919 podStartE2EDuration="5.656351596s" totalImagesPullingTime="4.154843677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:36.654799108 +0000 UTC m=+63.820126374" watchObservedRunningTime="2026-09-09 20:53:36.656351596 +0000 UTC m=+63.821678863"
Sep 09 20:53:37.154513 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.154478    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:53:37.156487 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.156468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.158938 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.158917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kgh55\""
Sep 09 20:53:37.160083 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.160065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:37.161339 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.161320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:37.161716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.161534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:37.161716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.161600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:37.161716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.161534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:37.165139 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.165122    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:37.175316 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.175284    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:53:37.253722 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.253872 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.253872 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.253872 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22t8\" (UniqueName: \"kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.253872 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.254048 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.254048 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.253922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354213 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354340 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354340 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354340 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354502 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354502 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22t8\" (UniqueName: \"kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.354502 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.354481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.355198 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.355142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.355299 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.355200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.355299 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.355203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.355823 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.355803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.357327 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.357304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.357432 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.357397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.364809 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.364784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22t8\" (UniqueName: \"kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8\") pod \"console-546846bdf8-lchdj\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") " pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.466888 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.466803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:37.596646 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.596620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kw5d2"
Sep 09 20:53:37.768068 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:37.767988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:53:37.772587 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:37.772557    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9095be14_ebc1_4b1a_a61e_da5c7d122386.slice/crio-3f56de1b6dc6ec31afc32ec2796dc15acd52dda3596c24ac1d513ecb7645adc1 WatchSource:0}: Error finding container 3f56de1b6dc6ec31afc32ec2796dc15acd52dda3596c24ac1d513ecb7645adc1: Status 404 returned error can't find the container with id 3f56de1b6dc6ec31afc32ec2796dc15acd52dda3596c24ac1d513ecb7645adc1
Sep 09 20:53:38.643682 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:38.643644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546846bdf8-lchdj" event={"ID":"9095be14-ebc1-4b1a-a61e-da5c7d122386","Type":"ContainerStarted","Data":"3f56de1b6dc6ec31afc32ec2796dc15acd52dda3596c24ac1d513ecb7645adc1"}
Sep 09 20:53:38.969599 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:38.969519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:38.972379 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:38.972354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:38.983383 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:38.983362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1b4883-3bbb-459e-b700-5a38c2131ba3-metrics-certs\") pod \"network-metrics-daemon-5k56s\" (UID: \"ee1b4883-3bbb-459e-b700-5a38c2131ba3\") " pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:38.995685 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:38.995657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pstzn\""
Sep 09 20:53:39.003108 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.003084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k56s"
Sep 09 20:53:39.070986 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.070770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:39.070986 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.070833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:39.075200 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.074957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:39.075658 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.075485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:39.092621 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.085275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d89900e-61d0-4ad4-9e29-8ffdc512f12b-original-pull-secret\") pod \"global-pull-secret-syncer-4wzrr\" (UID: \"2d89900e-61d0-4ad4-9e29-8ffdc512f12b\") " pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:39.092621 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.085456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:39.096179 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.095925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckvd\" (UniqueName: \"kubernetes.io/projected/14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5-kube-api-access-9ckvd\") pod \"network-check-target-9g5pw\" (UID: \"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5\") " pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:39.286914 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.286850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wzrr"
Sep 09 20:53:39.299632 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.299427    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-thpvv\""
Sep 09 20:53:39.306709 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.306687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:39.313361 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.313295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5k56s"]
Sep 09 20:53:39.317980 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:39.317958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1b4883_3bbb_459e_b700_5a38c2131ba3.slice/crio-9ecd86052829e93fb147b51a3f85856c077262181a3f0aa4c2b7471dbd7bec55 WatchSource:0}: Error finding container 9ecd86052829e93fb147b51a3f85856c077262181a3f0aa4c2b7471dbd7bec55: Status 404 returned error can't find the container with id 9ecd86052829e93fb147b51a3f85856c077262181a3f0aa4c2b7471dbd7bec55
Sep 09 20:53:39.605238 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.605175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wzrr"]
Sep 09 20:53:39.650007 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.649927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k56s" event={"ID":"ee1b4883-3bbb-459e-b700-5a38c2131ba3","Type":"ContainerStarted","Data":"9ecd86052829e93fb147b51a3f85856c077262181a3f0aa4c2b7471dbd7bec55"}
Sep 09 20:53:39.650525 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:39.650499    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9g5pw"]
Sep 09 20:53:40.774248 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:40.774214    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d89900e_61d0_4ad4_9e29_8ffdc512f12b.slice/crio-63279576da28b87016fe1e9799e1fce12ee154b39523bcd22a6631c65bc2a263 WatchSource:0}: Error finding container 63279576da28b87016fe1e9799e1fce12ee154b39523bcd22a6631c65bc2a263: Status 404 returned error can't find the container with id 63279576da28b87016fe1e9799e1fce12ee154b39523bcd22a6631c65bc2a263
Sep 09 20:53:40.775381 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:40.775349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fd5f9d_1f7d_4310_9326_06bb7c2c9bd5.slice/crio-96179af514b289ba4019f1c21662a3d678e96b3030693b00ff64d1d0fe9c55ee WatchSource:0}: Error finding container 96179af514b289ba4019f1c21662a3d678e96b3030693b00ff64d1d0fe9c55ee: Status 404 returned error can't find the container with id 96179af514b289ba4019f1c21662a3d678e96b3030693b00ff64d1d0fe9c55ee
Sep 09 20:53:41.661447 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.661371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k56s" event={"ID":"ee1b4883-3bbb-459e-b700-5a38c2131ba3","Type":"ContainerStarted","Data":"885768bfdfaddbef00728381cb824aa1b8e03f4002c1613742a578cc5aa0ac2c"}
Sep 09 20:53:41.661447 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.661426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k56s" event={"ID":"ee1b4883-3bbb-459e-b700-5a38c2131ba3","Type":"ContainerStarted","Data":"47eb47777661d5dfb6466acba84512bfa66fcb745a9935bdccaf55bf63f91ef8"}
Sep 09 20:53:41.664078 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.664047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546846bdf8-lchdj" event={"ID":"9095be14-ebc1-4b1a-a61e-da5c7d122386","Type":"ContainerStarted","Data":"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"}
Sep 09 20:53:41.666835 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.666781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9g5pw" event={"ID":"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5","Type":"ContainerStarted","Data":"96179af514b289ba4019f1c21662a3d678e96b3030693b00ff64d1d0fe9c55ee"}
Sep 09 20:53:41.668625 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.668604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wzrr" event={"ID":"2d89900e-61d0-4ad4-9e29-8ffdc512f12b","Type":"ContainerStarted","Data":"63279576da28b87016fe1e9799e1fce12ee154b39523bcd22a6631c65bc2a263"}
Sep 09 20:53:41.786570 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.786517    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5k56s" podStartSLOduration=66.729675377 podStartE2EDuration="1m8.786499665s" totalImagesPullingTime="2.056824288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:41.69379788 +0000 UTC m=+68.859125147" watchObservedRunningTime="2026-09-09 20:53:41.786499665 +0000 UTC m=+68.951826929"
Sep 09 20:53:41.787154 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:41.787114    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546846bdf8-lchdj" podStartSLOduration=1.7577783820000001 podStartE2EDuration="4.787098426s" totalImagesPullingTime="3.029320044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:41.785814643 +0000 UTC m=+68.951141912" watchObservedRunningTime="2026-09-09 20:53:41.787098426 +0000 UTC m=+68.952425692"
Sep 09 20:53:42.392354 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.391080    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-59lhg"]
Sep 09 20:53:42.394499 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.393542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.416094 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:53:42.416206 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416112    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:42.416206 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416067    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:42.416516 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416498    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:42.416632 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vrjqf\""
Sep 09 20:53:42.416714 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.416692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:53:42.417586 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.417554    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-wtmp\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-tls\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw5l\" (UniqueName: \"kubernetes.io/projected/087a2bcc-a7cd-4a95-b913-9318847cfb5d-kube-api-access-kgw5l\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-textfile\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-root\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-sys\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.504549 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.504266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-metrics-client-ca\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-root\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-sys\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-root\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-metrics-client-ca\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-wtmp\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-tls\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw5l\" (UniqueName: \"kubernetes.io/projected/087a2bcc-a7cd-4a95-b913-9318847cfb5d-kube-api-access-kgw5l\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.604961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-textfile\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.605344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-metrics-client-ca\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.605406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-sys\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.605575 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.605515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-wtmp\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.606957 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.606556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-textfile\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.606957 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.606919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.608723 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.608351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-tls\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.608723 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.608673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/087a2bcc-a7cd-4a95-b913-9318847cfb5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.622263 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.622241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw5l\" (UniqueName: \"kubernetes.io/projected/087a2bcc-a7cd-4a95-b913-9318847cfb5d-kube-api-access-kgw5l\") pod \"node-exporter-59lhg\" (UID: \"087a2bcc-a7cd-4a95-b913-9318847cfb5d\") " pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:42.706810 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:42.706563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59lhg"
Sep 09 20:53:45.524269 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.524235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"]
Sep 09 20:53:45.528061 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.528037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.533231 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.533208    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 20:53:45.533395 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.533295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 20:53:45.533546 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.533530    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 20:53:45.533629 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.533556    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-52qc7\""
Sep 09 20:53:45.534240 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.534216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 20:53:45.534342 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.534264    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-19dkuo413g251\""
Sep 09 20:53:45.534397 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.534222    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 20:53:45.549522 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.549500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"]
Sep 09 20:53:45.628960 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.628933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629116 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.628966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629116 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-metrics-client-ca\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629116 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629284 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtkf\" (UniqueName: \"kubernetes.io/projected/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-kube-api-access-fqtkf\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629284 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-grpc-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629284 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.629284 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.629254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.729837 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730004 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtkf\" (UniqueName: \"kubernetes.io/projected/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-kube-api-access-fqtkf\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730004 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-grpc-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730004 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730004 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730004 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.729995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730254 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.730028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.730254 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.730053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-metrics-client-ca\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.731053 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.730992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-metrics-client-ca\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.733048 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.733021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.733450 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.733426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.733542 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.733495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.733687 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.733666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.733774 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.733708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.734049 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.734033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-secret-grpc-tls\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.744998 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.744978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtkf\" (UniqueName: \"kubernetes.io/projected/8b88e86f-c45e-42d9-a9c3-52a8fbc964e0-kube-api-access-fqtkf\") pod \"thanos-querier-6bb486c4bf-hg6dx\" (UID: \"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0\") " pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:45.840622 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:45.840597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:46.982622 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:46.982581    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78588c9594-6hwfm"]
Sep 09 20:53:46.984803 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:46.984781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.007057 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.007033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 20:53:47.008263 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.008245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 20:53:47.008358 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.008307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1mdq4c7apnrbo\""
Sep 09 20:53:47.009903 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.009882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nks8q\""
Sep 09 20:53:47.014100 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.014081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 20:53:47.014232 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.014213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:53:47.031030 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.031004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78588c9594-6hwfm"]
Sep 09 20:53:47.042012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.041988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zmj\" (UniqueName: \"kubernetes.io/projected/b00090f2-28d4-4b54-a9a1-4f3c24094d99-kube-api-access-f9zmj\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-client-certs\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-client-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042125 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00090f2-28d4-4b54-a9a1-4f3c24094d99-audit-log\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042280 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-metrics-server-audit-profiles\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.042280 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.042187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-tls\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143334 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zmj\" (UniqueName: \"kubernetes.io/projected/b00090f2-28d4-4b54-a9a1-4f3c24094d99-kube-api-access-f9zmj\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143478 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-client-certs\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143478 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-client-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143478 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143666 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00090f2-28d4-4b54-a9a1-4f3c24094d99-audit-log\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143735 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-metrics-server-audit-profiles\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.143802 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.143787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-tls\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.144131 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.144106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00090f2-28d4-4b54-a9a1-4f3c24094d99-audit-log\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.144131 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.144110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.144779 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.144696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00090f2-28d4-4b54-a9a1-4f3c24094d99-metrics-server-audit-profiles\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.146366 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.146346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-client-certs\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.146635 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.146613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-secret-metrics-server-tls\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.147378 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.147357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00090f2-28d4-4b54-a9a1-4f3c24094d99-client-ca-bundle\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.156940 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.156916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zmj\" (UniqueName: \"kubernetes.io/projected/b00090f2-28d4-4b54-a9a1-4f3c24094d99-kube-api-access-f9zmj\") pod \"metrics-server-78588c9594-6hwfm\" (UID: \"b00090f2-28d4-4b54-a9a1-4f3c24094d99\") " pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.296354 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.296294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:53:47.467426 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.467386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:47.467597 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.467448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:47.474074 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.474048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:47.698377 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:47.698353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:53:50.407393 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.407353    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:53:50.439610 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.439583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:53:50.439774 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.439691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576691 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576691 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576929 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx429\" (UniqueName: \"kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576929 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576929 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.576929 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.577094 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.576930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678301 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678301 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx429\" (UniqueName: \"kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.678511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.678445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.679303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.679117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.679303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.679136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.679303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.679249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.679303 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.679288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.681247 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.681221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.681455 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.681432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.688687 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.688665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx429\" (UniqueName: \"kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429\") pod \"console-6b6454d865-b9f54\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") " pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:50.750044 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:50.749830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:53:52.832311 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:52.832029    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087a2bcc_a7cd_4a95_b913_9318847cfb5d.slice/crio-c9506e1477e3ebe4285006eee072f961ecf22f2b14f5ed57f5776e78adfdd421 WatchSource:0}: Error finding container c9506e1477e3ebe4285006eee072f961ecf22f2b14f5ed57f5776e78adfdd421: Status 404 returned error can't find the container with id c9506e1477e3ebe4285006eee072f961ecf22f2b14f5ed57f5776e78adfdd421
Sep 09 20:53:53.122699 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.122638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78588c9594-6hwfm"]
Sep 09 20:53:53.128852 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.128739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"]
Sep 09 20:53:53.128960 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.128907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:53:53.136834 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:53:53.135607    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b88e86f_c45e_42d9_a9c3_52a8fbc964e0.slice/crio-de84b7ea5e23bcef7c8d72ce4851dca33e42d89bdf9bd4b9a3ad456fbf2e3009 WatchSource:0}: Error finding container de84b7ea5e23bcef7c8d72ce4851dca33e42d89bdf9bd4b9a3ad456fbf2e3009: Status 404 returned error can't find the container with id de84b7ea5e23bcef7c8d72ce4851dca33e42d89bdf9bd4b9a3ad456fbf2e3009
Sep 09 20:53:53.718461 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.718419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"de84b7ea5e23bcef7c8d72ce4851dca33e42d89bdf9bd4b9a3ad456fbf2e3009"}
Sep 09 20:53:53.720911 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.720877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-zgc8d" event={"ID":"9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8","Type":"ContainerStarted","Data":"a06bf8a25e9bb7a5823af37c8f02b5072839acd3c5c9db1621bdb511dc727e8c"}
Sep 09 20:53:53.721999 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.721895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:53.727233 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.727198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6454d865-b9f54" event={"ID":"ec626cfb-c88f-45e0-836b-e7b497118dd6","Type":"ContainerStarted","Data":"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"}
Sep 09 20:53:53.727354 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.727250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6454d865-b9f54" event={"ID":"ec626cfb-c88f-45e0-836b-e7b497118dd6","Type":"ContainerStarted","Data":"c9a204242c604895282ec2e313b862e6a5f4da87ea255dd01b7ecb22ad000c02"}
Sep 09 20:53:53.731719 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.731233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9g5pw" event={"ID":"14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5","Type":"ContainerStarted","Data":"33e896c747cfa1c9cc06ba5a8a02e8da7b224e3152f0a24c8ef04b5254a8a807"}
Sep 09 20:53:53.731719 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.731685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:53:53.734493 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.734104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wzrr" event={"ID":"2d89900e-61d0-4ad4-9e29-8ffdc512f12b","Type":"ContainerStarted","Data":"5359f5e24ee374f3e7dbe3113874467fded98604d6a2a1e0c24568c9152ceae6"}
Sep 09 20:53:53.739801 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.739777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm" event={"ID":"b00090f2-28d4-4b54-a9a1-4f3c24094d99","Type":"ContainerStarted","Data":"2a30191b1eb7dd7aab56bd095bc96d13784159e96deaa2734ed70efc11081c45"}
Sep 09 20:53:53.740206 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.740187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-zgc8d"
Sep 09 20:53:53.741453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.741410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59lhg" event={"ID":"087a2bcc-a7cd-4a95-b913-9318847cfb5d","Type":"ContainerStarted","Data":"c9506e1477e3ebe4285006eee072f961ecf22f2b14f5ed57f5776e78adfdd421"}
Sep 09 20:53:53.786602 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.786538    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-zgc8d" podStartSLOduration=2.018731114 podStartE2EDuration="22.786518987s" totalImagesPullingTime="20.767787873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.753546936 +0000 UTC m=+80.918874204" watchObservedRunningTime="2026-09-09 20:53:53.786518987 +0000 UTC m=+80.951846297"
Sep 09 20:53:53.834295 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.834235    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9g5pw" podStartSLOduration=68.776401296 podStartE2EDuration="1m20.834213004s" totalImagesPullingTime="12.057811708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.827627708 +0000 UTC m=+80.992954976" watchObservedRunningTime="2026-09-09 20:53:53.834213004 +0000 UTC m=+80.999540272"
Sep 09 20:53:53.904532 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:53.904445    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6454d865-b9f54" podStartSLOduration=3.9044212959999998 podStartE2EDuration="3.904421296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.866738552 +0000 UTC m=+81.032065821" watchObservedRunningTime="2026-09-09 20:53:53.904421296 +0000 UTC m=+81.069748563"
Sep 09 20:53:55.753121 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:55.753082    2578 generic.go:356] "Generic (PLEG): container finished" podID="087a2bcc-a7cd-4a95-b913-9318847cfb5d" containerID="1d2aa3946b554bc82d2387262a95c596eb1408a5b04aac6a16ffd7fdd71aa5e2" exitCode=0
Sep 09 20:53:55.753544 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:55.753125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59lhg" event={"ID":"087a2bcc-a7cd-4a95-b913-9318847cfb5d","Type":"ContainerDied","Data":"1d2aa3946b554bc82d2387262a95c596eb1408a5b04aac6a16ffd7fdd71aa5e2"}
Sep 09 20:53:55.820218 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:55.820170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4wzrr" podStartSLOduration=70.761857752 podStartE2EDuration="1m22.820154133s" totalImagesPullingTime="12.058296381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.904889094 +0000 UTC m=+81.070216374" watchObservedRunningTime="2026-09-09 20:53:55.820154133 +0000 UTC m=+82.985481399"
Sep 09 20:53:56.759926 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:56.759889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm" event={"ID":"b00090f2-28d4-4b54-a9a1-4f3c24094d99","Type":"ContainerStarted","Data":"10ac960198939300147b8bca615524eaa71a1109529423a95d8ae7f9725dc110"}
Sep 09 20:53:56.762679 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:56.762654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59lhg" event={"ID":"087a2bcc-a7cd-4a95-b913-9318847cfb5d","Type":"ContainerStarted","Data":"0c96b1905fb2bd4abf3b941270ba57545d65c5669a5798c4900e2ca6de8a9779"}
Sep 09 20:53:56.762782 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:56.762707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59lhg" event={"ID":"087a2bcc-a7cd-4a95-b913-9318847cfb5d","Type":"ContainerStarted","Data":"481755dd19c70148568fa21101e61c4886d4edfb87bc678cece79641defd8566"}
Sep 09 20:53:56.819460 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:56.819413    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm" podStartSLOduration=7.719887648 podStartE2EDuration="10.819393359s" totalImagesPullingTime="3.099505711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:56.818215635 +0000 UTC m=+83.983542902" watchObservedRunningTime="2026-09-09 20:53:56.819393359 +0000 UTC m=+83.984720627"
Sep 09 20:53:57.770899 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:57.770816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"2bbd26bd28f357187e442724d5ea452e3e9fcc3fcb6326112c6126fdf451cb1c"}
Sep 09 20:53:57.770899 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:57.770903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"e8eb0ab5bad12e957f945f5d165a951b9467ce3543754175576fa7c698237712"}
Sep 09 20:53:57.771641 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:57.770944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"fee870870b5abe758f67d00198b82e0f314997df963ef34ee30249c0667ca9cc"}
Sep 09 20:53:58.778621 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:58.778587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"7153036bea8b59c6e9c755372aa3a2c9ad6e12cf947a2c0b52e61dbcebf3963c"}
Sep 09 20:53:58.779000 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:58.778672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"d508fac48e703f806131b05de057c50cf823c72f73d588551390381cfa1e5243"}
Sep 09 20:53:59.786538 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:59.786495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" event={"ID":"8b88e86f-c45e-42d9-a9c3-52a8fbc964e0","Type":"ContainerStarted","Data":"bca0dce3f959b4e16073b436b16da76cf2aa0d83c61e35a0fdb70d8ed70d3989"}
Sep 09 20:53:59.787119 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:59.786655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:53:59.835417 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:59.835363    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-59lhg" podStartSLOduration=15.318691462 podStartE2EDuration="17.835347545s" totalImagesPullingTime="2.403260275s" totalInitContainerRuntime="113.395808ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:56.861966085 +0000 UTC m=+84.027293365" watchObservedRunningTime="2026-09-09 20:53:59.835347545 +0000 UTC m=+87.000674812"
Sep 09 20:53:59.836232 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:53:59.836190    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx" podStartSLOduration=9.888012575 podStartE2EDuration="14.836179793s" totalImagesPullingTime="4.948167218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:59.833167363 +0000 UTC m=+86.998494632" watchObservedRunningTime="2026-09-09 20:53:59.836179793 +0000 UTC m=+87.001507061"
Sep 09 20:54:00.750241 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:00.750205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:54:00.750499 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:00.750276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:54:00.755692 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:00.755660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:54:00.795537 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:00.795509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:54:00.898137 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:00.898104    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:54:05.797353 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:05.797322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bb486c4bf-hg6dx"
Sep 09 20:54:07.297362 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:07.297331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:54:07.297825 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:07.297382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:54:25.755798 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:25.755767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9g5pw"
Sep 09 20:54:25.925603 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:25.925566    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546846bdf8-lchdj" podUID="9095be14-ebc1-4b1a-a61e-da5c7d122386" containerName="console" containerID="cri-o://53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159" gracePeriod=15
Sep 09 20:54:26.191445 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.191418    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:54:26.242787 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.242787 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243013 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243013 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243013 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f22t8\" (UniqueName: \"kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243013 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243013 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.242937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert\") pod \"9095be14-ebc1-4b1a-a61e-da5c7d122386\" (UID: \"9095be14-ebc1-4b1a-a61e-da5c7d122386\") "
Sep 09 20:54:26.243381 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.243350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config" (OuterVolumeSpecName: "console-config") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:26.243470 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.243377    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:26.243470 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.243358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca" (OuterVolumeSpecName: "service-ca") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:26.243470 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.243442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:26.245418 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.245381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:26.245511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.245431    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:26.245511 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.245430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8" (OuterVolumeSpecName: "kube-api-access-f22t8") pod "9095be14-ebc1-4b1a-a61e-da5c7d122386" (UID: "9095be14-ebc1-4b1a-a61e-da5c7d122386"). InnerVolumeSpecName "kube-api-access-f22t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:26.343870 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343844    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.343870 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343871    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22t8\" (UniqueName: \"kubernetes.io/projected/9095be14-ebc1-4b1a-a61e-da5c7d122386-kube-api-access-f22t8\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.344012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343885    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-serving-cert\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.344012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343899    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-oauth-serving-cert\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.344012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343911    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9095be14-ebc1-4b1a-a61e-da5c7d122386-console-oauth-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.344012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343924    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-service-ca\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.344012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.343936    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9095be14-ebc1-4b1a-a61e-da5c7d122386-trusted-ca-bundle\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:54:26.883605 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.883570    2578 generic.go:356] "Generic (PLEG): container finished" podID="9095be14-ebc1-4b1a-a61e-da5c7d122386" containerID="53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159" exitCode=0
Sep 09 20:54:26.884033 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.883619    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546846bdf8-lchdj"
Sep 09 20:54:26.884033 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.883650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546846bdf8-lchdj" event={"ID":"9095be14-ebc1-4b1a-a61e-da5c7d122386","Type":"ContainerDied","Data":"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"}
Sep 09 20:54:26.884033 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.883701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546846bdf8-lchdj" event={"ID":"9095be14-ebc1-4b1a-a61e-da5c7d122386","Type":"ContainerDied","Data":"3f56de1b6dc6ec31afc32ec2796dc15acd52dda3596c24ac1d513ecb7645adc1"}
Sep 09 20:54:26.884033 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.883721    2578 scope.go:117] "RemoveContainer" containerID="53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"
Sep 09 20:54:26.894958 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.894944    2578 scope.go:117] "RemoveContainer" containerID="53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"
Sep 09 20:54:26.895242 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:54:26.895216    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159\": container with ID starting with 53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159 not found: ID does not exist" containerID="53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"
Sep 09 20:54:26.895291 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.895254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159"} err="failed to get container status \"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159\": rpc error: code = NotFound desc = could not find container \"53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159\": container with ID starting with 53697c880464a68d7274da3c901efa6d45a2e39164a2a0bb5907954a53358159 not found: ID does not exist"
Sep 09 20:54:26.910221 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.910199    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:54:26.912894 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:26.912876    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546846bdf8-lchdj"]
Sep 09 20:54:27.281721 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:27.281649    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9095be14-ebc1-4b1a-a61e-da5c7d122386" path="/var/lib/kubelet/pods/9095be14-ebc1-4b1a-a61e-da5c7d122386/volumes"
Sep 09 20:54:27.302072 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:27.302044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:54:27.305703 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:54:27.305676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78588c9594-6hwfm"
Sep 09 20:55:20.634258 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:20.634215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:55:45.653257 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.653156    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b6454d865-b9f54" podUID="ec626cfb-c88f-45e0-836b-e7b497118dd6" containerName="console" containerID="cri-o://14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7" gracePeriod=15
Sep 09 20:55:45.892370 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.892343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:55:45.987474 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987474 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx429\" (UniqueName: \"kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987474 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987682 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987682 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987682 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.987846 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert\") pod \"ec626cfb-c88f-45e0-836b-e7b497118dd6\" (UID: \"ec626cfb-c88f-45e0-836b-e7b497118dd6\") "
Sep 09 20:55:45.988005 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.987959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:45.988129 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.988082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config" (OuterVolumeSpecName: "console-config") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:45.988129 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.988097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:45.988415 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.988135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:45.989928 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.989905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429" (OuterVolumeSpecName: "kube-api-access-tx429") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "kube-api-access-tx429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:45.990060 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.990032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:45.990101 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:45.990080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec626cfb-c88f-45e0-836b-e7b497118dd6" (UID: "ec626cfb-c88f-45e0-836b-e7b497118dd6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:46.089020 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089000    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-service-ca\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089020 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089019    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-oauth-serving-cert\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089126 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089028    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089126 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089037    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-serving-cert\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089126 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089046    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec626cfb-c88f-45e0-836b-e7b497118dd6-trusted-ca-bundle\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089126 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089054    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx429\" (UniqueName: \"kubernetes.io/projected/ec626cfb-c88f-45e0-836b-e7b497118dd6-kube-api-access-tx429\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.089126 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.089062    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec626cfb-c88f-45e0-836b-e7b497118dd6-console-oauth-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:55:46.166832 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.166803    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec626cfb-c88f-45e0-836b-e7b497118dd6" containerID="14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7" exitCode=0
Sep 09 20:55:46.166912 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.166854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6454d865-b9f54"
Sep 09 20:55:46.166954 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.166851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6454d865-b9f54" event={"ID":"ec626cfb-c88f-45e0-836b-e7b497118dd6","Type":"ContainerDied","Data":"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"}
Sep 09 20:55:46.166988 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.166972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6454d865-b9f54" event={"ID":"ec626cfb-c88f-45e0-836b-e7b497118dd6","Type":"ContainerDied","Data":"c9a204242c604895282ec2e313b862e6a5f4da87ea255dd01b7ecb22ad000c02"}
Sep 09 20:55:46.167030 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.167000    2578 scope.go:117] "RemoveContainer" containerID="14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"
Sep 09 20:55:46.175334 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.175316    2578 scope.go:117] "RemoveContainer" containerID="14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"
Sep 09 20:55:46.175594 ip-10-0-142-161 kubenswrapper[2578]: E0909 20:55:46.175576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7\": container with ID starting with 14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7 not found: ID does not exist" containerID="14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"
Sep 09 20:55:46.175660 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.175601    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7"} err="failed to get container status \"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7\": rpc error: code = NotFound desc = could not find container \"14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7\": container with ID starting with 14471e250c2f7001678d2ef98bd27895a08e2fe63d26c9cbe03f494be8661ed7 not found: ID does not exist"
Sep 09 20:55:46.189788 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.189768    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:55:46.193781 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:46.193764    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6454d865-b9f54"]
Sep 09 20:55:47.280678 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:55:47.280650    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec626cfb-c88f-45e0-836b-e7b497118dd6" path="/var/lib/kubelet/pods/ec626cfb-c88f-45e0-836b-e7b497118dd6/volumes"
Sep 09 20:56:28.752843 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.752812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"]
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753061    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9095be14-ebc1-4b1a-a61e-da5c7d122386" containerName="console"
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753071    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9095be14-ebc1-4b1a-a61e-da5c7d122386" containerName="console"
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753084    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec626cfb-c88f-45e0-836b-e7b497118dd6" containerName="console"
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753089    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec626cfb-c88f-45e0-836b-e7b497118dd6" containerName="console"
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753136    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9095be14-ebc1-4b1a-a61e-da5c7d122386" containerName="console"
Sep 09 20:56:28.753276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.753145    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec626cfb-c88f-45e0-836b-e7b497118dd6" containerName="console"
Sep 09 20:56:28.754917 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.754902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.758770 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.758737    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:56:28.758900 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.758777    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zllrt\""
Sep 09 20:56:28.759704 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.759689    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:56:28.764238 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.764220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.764323 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.764248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.764323 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.764268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6z9\" (UniqueName: \"kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.772137 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.772118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"]
Sep 09 20:56:28.864844 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.864818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.864964 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.864849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.864964 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.864868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6z9\" (UniqueName: \"kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.865164 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.865147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.865229 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.865208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:28.873377 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:28.873353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6z9\" (UniqueName: \"kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:29.064166 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:29.064078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:29.332735 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:29.332708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"]
Sep 09 20:56:29.335086 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:56:29.335057    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc794f361_5cc6_4ecf_a6cb_60be7e05fc91.slice/crio-1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d WatchSource:0}: Error finding container 1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d: Status 404 returned error can't find the container with id 1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d
Sep 09 20:56:30.315420 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:30.315382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh" event={"ID":"c794f361-5cc6-4ecf-a6cb-60be7e05fc91","Type":"ContainerStarted","Data":"1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d"}
Sep 09 20:56:35.334650 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:35.334612    2578 generic.go:356] "Generic (PLEG): container finished" podID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerID="a726b9ea3c441f0859d632ca977f09c5e40847d2d3cb808c2947cf4a4260ac76" exitCode=0
Sep 09 20:56:35.335136 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:35.334694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh" event={"ID":"c794f361-5cc6-4ecf-a6cb-60be7e05fc91","Type":"ContainerDied","Data":"a726b9ea3c441f0859d632ca977f09c5e40847d2d3cb808c2947cf4a4260ac76"}
Sep 09 20:56:37.344166 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:37.344141    2578 generic.go:356] "Generic (PLEG): container finished" podID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerID="6c2b20979610563b9e8709275d6579771bd02fb7db336bda4beedd55c73cbc35" exitCode=0
Sep 09 20:56:37.344476 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:37.344221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh" event={"ID":"c794f361-5cc6-4ecf-a6cb-60be7e05fc91","Type":"ContainerDied","Data":"6c2b20979610563b9e8709275d6579771bd02fb7db336bda4beedd55c73cbc35"}
Sep 09 20:56:44.375622 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:44.375584    2578 generic.go:356] "Generic (PLEG): container finished" podID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerID="ec7cfd19460a7c37def459384961427152f6478ac5e5fa9f3b191c9d4a565c53" exitCode=0
Sep 09 20:56:44.375989 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:44.375663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh" event={"ID":"c794f361-5cc6-4ecf-a6cb-60be7e05fc91","Type":"ContainerDied","Data":"ec7cfd19460a7c37def459384961427152f6478ac5e5fa9f3b191c9d4a565c53"}
Sep 09 20:56:45.499422 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.499401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:45.586857 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.586834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle\") pod \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") "
Sep 09 20:56:45.586998 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.586876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util\") pod \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") "
Sep 09 20:56:45.586998 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.586894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt6z9\" (UniqueName: \"kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9\") pod \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\" (UID: \"c794f361-5cc6-4ecf-a6cb-60be7e05fc91\") "
Sep 09 20:56:45.587449 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.587426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle" (OuterVolumeSpecName: "bundle") pod "c794f361-5cc6-4ecf-a6cb-60be7e05fc91" (UID: "c794f361-5cc6-4ecf-a6cb-60be7e05fc91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.589028 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.589007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9" (OuterVolumeSpecName: "kube-api-access-pt6z9") pod "c794f361-5cc6-4ecf-a6cb-60be7e05fc91" (UID: "c794f361-5cc6-4ecf-a6cb-60be7e05fc91"). InnerVolumeSpecName "kube-api-access-pt6z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:45.591576 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.591543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util" (OuterVolumeSpecName: "util") pod "c794f361-5cc6-4ecf-a6cb-60be7e05fc91" (UID: "c794f361-5cc6-4ecf-a6cb-60be7e05fc91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.687761 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.687681    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-bundle\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.687761 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.687705    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-util\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.687761 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:45.687717    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6z9\" (UniqueName: \"kubernetes.io/projected/c794f361-5cc6-4ecf-a6cb-60be7e05fc91-kube-api-access-pt6z9\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 20:56:46.384718 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:46.384682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh" event={"ID":"c794f361-5cc6-4ecf-a6cb-60be7e05fc91","Type":"ContainerDied","Data":"1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d"}
Sep 09 20:56:46.384871 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:46.384720    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ntjh"
Sep 09 20:56:46.384871 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:46.384735    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3adba6aaea1be92a3f5961f5539d235740dde10147386b898a301013d47e1d"
Sep 09 20:56:52.346077 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346041    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"]
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346257    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="util"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346268    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="util"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346278    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="extract"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="extract"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346295    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="pull"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346300    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="pull"
Sep 09 20:56:52.346453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.346344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c794f361-5cc6-4ecf-a6cb-60be7e05fc91" containerName="extract"
Sep 09 20:56:52.348282 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.348267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.351478 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.351455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8p2lb\""
Sep 09 20:56:52.351608 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.351568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:56:52.351843 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.351825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:56:52.351919 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.351905    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:56:52.371479 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.371460    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"]
Sep 09 20:56:52.432335 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.432304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b5a2752-e55f-44d6-a25c-e8c63d886e53-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.432450 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.432369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkb5\" (UniqueName: \"kubernetes.io/projected/9b5a2752-e55f-44d6-a25c-e8c63d886e53-kube-api-access-qqkb5\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.533053 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.533028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b5a2752-e55f-44d6-a25c-e8c63d886e53-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.533156 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.533070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkb5\" (UniqueName: \"kubernetes.io/projected/9b5a2752-e55f-44d6-a25c-e8c63d886e53-kube-api-access-qqkb5\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.535449 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.535427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b5a2752-e55f-44d6-a25c-e8c63d886e53-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.542454 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.542434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkb5\" (UniqueName: \"kubernetes.io/projected/9b5a2752-e55f-44d6-a25c-e8c63d886e53-kube-api-access-qqkb5\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fdqfx\" (UID: \"9b5a2752-e55f-44d6-a25c-e8c63d886e53\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.658467 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.658400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:52.933200 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:52.933175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"]
Sep 09 20:56:52.934866 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:56:52.934840    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5a2752_e55f_44d6_a25c_e8c63d886e53.slice/crio-9872726ca7f22a0035c7811818da8ae07de33dc720dd45b26a7c885bf197c137 WatchSource:0}: Error finding container 9872726ca7f22a0035c7811818da8ae07de33dc720dd45b26a7c885bf197c137: Status 404 returned error can't find the container with id 9872726ca7f22a0035c7811818da8ae07de33dc720dd45b26a7c885bf197c137
Sep 09 20:56:53.407358 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:53.407329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx" event={"ID":"9b5a2752-e55f-44d6-a25c-e8c63d886e53","Type":"ContainerStarted","Data":"9872726ca7f22a0035c7811818da8ae07de33dc720dd45b26a7c885bf197c137"}
Sep 09 20:56:56.421117 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:56.421080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx" event={"ID":"9b5a2752-e55f-44d6-a25c-e8c63d886e53","Type":"ContainerStarted","Data":"db1034a91908851e3afab20a6f6a776541b5ef4a0fe54f047802bf2ec578554d"}
Sep 09 20:56:56.421548 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:56.421206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:56:56.469195 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:56.469150    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx" podStartSLOduration=1.4840030739999999 podStartE2EDuration="4.469138385s" totalImagesPullingTime="2.985135311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:56.468507724 +0000 UTC m=+263.633834992" watchObservedRunningTime="2026-09-09 20:56:56.469138385 +0000 UTC m=+263.634465652"
Sep 09 20:56:57.374093 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.374057    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"]
Sep 09 20:56:57.376298 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.376277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.382123 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.382100    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:56:57.382240 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.382108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:56:57.382414 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.382400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-v6g59\""
Sep 09 20:56:57.390220 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.390199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"]
Sep 09 20:56:57.468399 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.468372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-certificates\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.468739 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.468409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7kk\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-kube-api-access-jn7kk\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.468739 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.468470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a85bcad4-97bb-4bb2-83fa-04288ef74046-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.569620 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.569591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-certificates\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.569813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.569627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7kk\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-kube-api-access-jn7kk\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.569813 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.569648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a85bcad4-97bb-4bb2-83fa-04288ef74046-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.570012 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.569996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a85bcad4-97bb-4bb2-83fa-04288ef74046-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.572237 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.572214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-certificates\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.583604 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.583578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7kk\" (UniqueName: \"kubernetes.io/projected/a85bcad4-97bb-4bb2-83fa-04288ef74046-kube-api-access-jn7kk\") pod \"keda-metrics-apiserver-77857b7b98-h7xxs\" (UID: \"a85bcad4-97bb-4bb2-83fa-04288ef74046\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.686454 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.686391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:56:57.956554 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:57.956380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"]
Sep 09 20:56:57.959357 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:56:57.959333    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85bcad4_97bb_4bb2_83fa_04288ef74046.slice/crio-9b2a3bad7e4fc790017123bf895477efd4a502d22f622404cc22f9a62fda3467 WatchSource:0}: Error finding container 9b2a3bad7e4fc790017123bf895477efd4a502d22f622404cc22f9a62fda3467: Status 404 returned error can't find the container with id 9b2a3bad7e4fc790017123bf895477efd4a502d22f622404cc22f9a62fda3467
Sep 09 20:56:58.430400 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:56:58.430365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs" event={"ID":"a85bcad4-97bb-4bb2-83fa-04288ef74046","Type":"ContainerStarted","Data":"9b2a3bad7e4fc790017123bf895477efd4a502d22f622404cc22f9a62fda3467"}
Sep 09 20:57:00.441172 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:00.441139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs" event={"ID":"a85bcad4-97bb-4bb2-83fa-04288ef74046","Type":"ContainerStarted","Data":"0dceb0c4435417bf7ec1e33cc4e2a2ff6bf2bea64d244632e5d17187d3eb750a"}
Sep 09 20:57:00.441570 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:00.441258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:57:00.460328 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:00.460265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs" podStartSLOduration=1.39187391 podStartE2EDuration="3.460246701s" totalImagesPullingTime="2.068372791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:00.458984047 +0000 UTC m=+267.624311316" watchObservedRunningTime="2026-09-09 20:57:00.460246701 +0000 UTC m=+267.625573972"
Sep 09 20:57:11.449453 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:11.449428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-h7xxs"
Sep 09 20:57:17.427678 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:17.427641    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fdqfx"
Sep 09 20:57:33.173565 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:57:33.173539    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:58:18.514640 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.514601    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"]
Sep 09 20:58:18.517170 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.517148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.520094 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.520075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:58:18.520208 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.520112    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:58:18.521141 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.521125    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:58:18.521425 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.521406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-df65q\""
Sep 09 20:58:18.537912 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.537879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"]
Sep 09 20:58:18.538206 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.538184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/816429ee-14e0-4fa0-8506-d999ffc0ac26-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.538270 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.538247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xksg\" (UniqueName: \"kubernetes.io/projected/816429ee-14e0-4fa0-8506-d999ffc0ac26-kube-api-access-8xksg\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.639325 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.639297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/816429ee-14e0-4fa0-8506-d999ffc0ac26-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.639473 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.639346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xksg\" (UniqueName: \"kubernetes.io/projected/816429ee-14e0-4fa0-8506-d999ffc0ac26-kube-api-access-8xksg\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.641831 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.641805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/816429ee-14e0-4fa0-8506-d999ffc0ac26-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.648818 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.648797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xksg\" (UniqueName: \"kubernetes.io/projected/816429ee-14e0-4fa0-8506-d999ffc0ac26-kube-api-access-8xksg\") pod \"llmisvc-controller-manager-645f9c8fd6-qj2kw\" (UID: \"816429ee-14e0-4fa0-8506-d999ffc0ac26\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:18.826525 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:18.826506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:19.098878 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:19.098812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"]
Sep 09 20:58:19.101963 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:58:19.101936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod816429ee_14e0_4fa0_8506_d999ffc0ac26.slice/crio-9010e6721db48fc33d83485d15084d53fe2c7956299feb5a325242d63c2ded7d WatchSource:0}: Error finding container 9010e6721db48fc33d83485d15084d53fe2c7956299feb5a325242d63c2ded7d: Status 404 returned error can't find the container with id 9010e6721db48fc33d83485d15084d53fe2c7956299feb5a325242d63c2ded7d
Sep 09 20:58:19.103251 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:19.103232    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:19.725557 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:19.725518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw" event={"ID":"816429ee-14e0-4fa0-8506-d999ffc0ac26","Type":"ContainerStarted","Data":"9010e6721db48fc33d83485d15084d53fe2c7956299feb5a325242d63c2ded7d"}
Sep 09 20:58:21.734302 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:21.734267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw" event={"ID":"816429ee-14e0-4fa0-8506-d999ffc0ac26","Type":"ContainerStarted","Data":"04295a7374e9fa70b29cf563a4d41b1652b4738d631074c3a0c1ef696dec4f00"}
Sep 09 20:58:21.734726 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:21.734377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 20:58:21.757716 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:21.757672    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw" podStartSLOduration=1.904486114 podStartE2EDuration="3.757658767s" totalImagesPullingTime="1.853172653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:21.757424918 +0000 UTC m=+348.922752187" watchObservedRunningTime="2026-09-09 20:58:21.757658767 +0000 UTC m=+348.922986034"
Sep 09 20:58:37.324734 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.324659    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c764c7d8d-5s7g2"]
Sep 09 20:58:37.326722 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.326707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.329884 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.329858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:58:37.330375 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.330349    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:58:37.331049 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.331032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:58:37.331571 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.331549    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:58:37.331571 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.331558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kgh55\""
Sep 09 20:58:37.331727 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.331591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:58:37.335639 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.335622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:58:37.348999 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.348979    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c764c7d8d-5s7g2"]
Sep 09 20:58:37.369863 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.369840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-service-ca\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.369950 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.369868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-trusted-ca-bundle\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.369950 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.369887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-oauth-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.369950 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.369912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-oauth-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.370053 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.369985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mfm\" (UniqueName: \"kubernetes.io/projected/50599695-bf36-40ab-abb1-b490abbd407c-kube-api-access-p8mfm\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.370053 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.370034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-console-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.370112 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.370057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471025 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.470997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-service-ca\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471123 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-trusted-ca-bundle\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471123 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-oauth-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471123 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-oauth-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471299 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mfm\" (UniqueName: \"kubernetes.io/projected/50599695-bf36-40ab-abb1-b490abbd407c-kube-api-access-p8mfm\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471299 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-console-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471299 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471764 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-service-ca\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.471971 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.471952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-console-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.472048 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.472029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-oauth-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.472140 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.472123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50599695-bf36-40ab-abb1-b490abbd407c-trusted-ca-bundle\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.473635 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.473610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-oauth-config\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.473707 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.473695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50599695-bf36-40ab-abb1-b490abbd407c-console-serving-cert\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.480316 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.480297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mfm\" (UniqueName: \"kubernetes.io/projected/50599695-bf36-40ab-abb1-b490abbd407c-kube-api-access-p8mfm\") pod \"console-5c764c7d8d-5s7g2\" (UID: \"50599695-bf36-40ab-abb1-b490abbd407c\") " pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.636515 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.636455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:37.909020 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:37.908929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c764c7d8d-5s7g2"]
Sep 09 20:58:37.911355 ip-10-0-142-161 kubenswrapper[2578]: W0909 20:58:37.911330    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50599695_bf36_40ab_abb1_b490abbd407c.slice/crio-4487354427b12c5b35d6c44b778646f15c830127cf550c33b44b5348036664ca WatchSource:0}: Error finding container 4487354427b12c5b35d6c44b778646f15c830127cf550c33b44b5348036664ca: Status 404 returned error can't find the container with id 4487354427b12c5b35d6c44b778646f15c830127cf550c33b44b5348036664ca
Sep 09 20:58:38.796518 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:38.796487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c764c7d8d-5s7g2" event={"ID":"50599695-bf36-40ab-abb1-b490abbd407c","Type":"ContainerStarted","Data":"d58f39909712bfe98fcaa0b668a8202647c994603f77e40585db7228f0ca031d"}
Sep 09 20:58:38.796872 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:38.796532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c764c7d8d-5s7g2" event={"ID":"50599695-bf36-40ab-abb1-b490abbd407c","Type":"ContainerStarted","Data":"4487354427b12c5b35d6c44b778646f15c830127cf550c33b44b5348036664ca"}
Sep 09 20:58:47.637276 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:47.637247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:47.637721 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:47.637328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:47.641712 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:47.641695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:47.675400 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:47.675355    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c764c7d8d-5s7g2" podStartSLOduration=10.67534086 podStartE2EDuration="10.67534086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:38.828634269 +0000 UTC m=+365.993961536" watchObservedRunningTime="2026-09-09 20:58:47.67534086 +0000 UTC m=+374.840668129"
Sep 09 20:58:47.832629 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:47.832607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c764c7d8d-5s7g2"
Sep 09 20:58:52.740670 ip-10-0-142-161 kubenswrapper[2578]: I0909 20:58:52.740638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-qj2kw"
Sep 09 21:01:23.983268 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.983211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:01:23.985931 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.985916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:23.990285 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.990261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:01:23.990736 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.990714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:01:23.990872 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.990714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88167-predictor-serving-cert\""
Sep 09 21:01:23.990872 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.990770    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88167-kube-rbac-proxy-sar-config\""
Sep 09 21:01:23.990872 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:23.990778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c7qlq\""
Sep 09 21:01:24.001147 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.001124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:01:24.104103 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.104078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.104206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.104115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkr8z\" (UniqueName: \"kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.104206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.104139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.204488 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.204461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.204586 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.204514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.204586 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.204541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkr8z\" (UniqueName: \"kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.204694 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:01:24.204675    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-88167-predictor-serving-cert: secret "success-200-isvc-88167-predictor-serving-cert" not found
Sep 09 21:01:24.204799 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:01:24.204787    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls podName:03003a0f-4a04-4164-ab6a-f522ddbe9bf4 nodeName:}" failed. No retries permitted until 2026-09-09 21:01:24.704762943 +0000 UTC m=+531.870090204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls") pod "success-200-isvc-88167-predictor-57db869b7c-f4qxf" (UID: "03003a0f-4a04-4164-ab6a-f522ddbe9bf4") : secret "success-200-isvc-88167-predictor-serving-cert" not found
Sep 09 21:01:24.205193 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.205177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.215006 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.214979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkr8z\" (UniqueName: \"kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.576709 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.576628    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:01:24.578873 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.578857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.581706 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.581682    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-88167-kube-rbac-proxy-sar-config\""
Sep 09 21:01:24.581842 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.581720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-88167-predictor-serving-cert\""
Sep 09 21:01:24.592188 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.592167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:01:24.708679 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.708648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.708877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.708767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.708877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.708802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.708877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.708834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6khc\" (UniqueName: \"kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.711345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.711324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") pod \"success-200-isvc-88167-predictor-57db869b7c-f4qxf\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") " pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:24.809397 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.809364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6khc\" (UniqueName: \"kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.809513 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.809418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.809513 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.809490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.809619 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:01:24.809595    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-88167-predictor-serving-cert: secret "error-404-isvc-88167-predictor-serving-cert" not found
Sep 09 21:01:24.809679 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:01:24.809670    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls podName:66f01940-469b-4063-bce9-07fbe8661e8a nodeName:}" failed. No retries permitted until 2026-09-09 21:01:25.309650721 +0000 UTC m=+532.474977973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls") pod "error-404-isvc-88167-predictor-df799f49b-g77jh" (UID: "66f01940-469b-4063-bce9-07fbe8661e8a") : secret "error-404-isvc-88167-predictor-serving-cert" not found
Sep 09 21:01:24.810196 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.810175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.820937 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.820909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6khc\" (UniqueName: \"kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:24.895779 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:24.895701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:25.170194 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.170173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:01:25.171852 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:01:25.171826    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03003a0f_4a04_4164_ab6a_f522ddbe9bf4.slice/crio-234fdb84104de0161935e8fa8026e896329baadbe4a7a1daabb6132a5b2cd8a2 WatchSource:0}: Error finding container 234fdb84104de0161935e8fa8026e896329baadbe4a7a1daabb6132a5b2cd8a2: Status 404 returned error can't find the container with id 234fdb84104de0161935e8fa8026e896329baadbe4a7a1daabb6132a5b2cd8a2
Sep 09 21:01:25.311998 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.311972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:25.314288 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.314266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") pod \"error-404-isvc-88167-predictor-df799f49b-g77jh\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") " pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:25.397887 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.397858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerStarted","Data":"234fdb84104de0161935e8fa8026e896329baadbe4a7a1daabb6132a5b2cd8a2"}
Sep 09 21:01:25.490931 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.490870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:25.774919 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:25.774891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:01:26.405365 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:26.405328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerStarted","Data":"3a8890fd62d2ce4e8e52712ed41a10f64705f06e45d89e7855f10248255ca80e"}
Sep 09 21:01:41.487275 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:41.487229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerStarted","Data":"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"}
Sep 09 21:01:41.488676 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:41.488653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerStarted","Data":"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"}
Sep 09 21:01:45.510262 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.510218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerStarted","Data":"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"}
Sep 09 21:01:45.510705 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.510288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:45.510705 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.510315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:45.512018 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.511983    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:01:45.512260 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.512237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerStarted","Data":"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"}
Sep 09 21:01:45.512422 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.512406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:45.512502 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.512455    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:45.513365 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.513345    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:01:45.529541 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.529502    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podStartSLOduration=3.230156809 podStartE2EDuration="22.529489007s" totalImagesPullingTime="19.299332198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:45.528947863 +0000 UTC m=+552.694275155" watchObservedRunningTime="2026-09-09 21:01:45.529489007 +0000 UTC m=+552.694816285"
Sep 09 21:01:45.548380 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:45.548334    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podStartSLOduration=2.84877299 podStartE2EDuration="21.548321467s" totalImagesPullingTime="18.699548477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:45.547647594 +0000 UTC m=+552.712974862" watchObservedRunningTime="2026-09-09 21:01:45.548321467 +0000 UTC m=+552.713648734"
Sep 09 21:01:46.516221 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:46.516176    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:01:46.516581 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:46.516255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:01:51.520891 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:51.520865    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:01:51.521312 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:51.521283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:01:51.521379 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:51.521353    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:01:51.521768 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:01:51.521724    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:02:01.521607 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:01.521567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:02:01.522073 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:01.521716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:02:11.521282 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:11.521245    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:02:11.521678 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:11.521641    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:02:21.521490 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:21.521448    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 21:02:21.521956 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:21.521687    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 21:02:31.521765 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:31.521717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:02:31.522434 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:31.522414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:02:54.488574 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.488536    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:02:54.489178 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.488818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container" containerID="cri-o://6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79" gracePeriod=30
Sep 09 21:02:54.489178 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.488882    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kube-rbac-proxy" containerID="cri-o://cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb" gracePeriod=30
Sep 09 21:02:54.555338 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.555308    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:02:54.555545 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.555509    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container" containerID="cri-o://7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d" gracePeriod=30
Sep 09 21:02:54.555618 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.555532    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kube-rbac-proxy" containerID="cri-o://9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3" gracePeriod=30
Sep 09 21:02:54.753308 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.753235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:02:54.755446 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.755429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.761644 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.761450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67ffc-kube-rbac-proxy-sar-config\""
Sep 09 21:02:54.761897 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.761880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67ffc-predictor-serving-cert\""
Sep 09 21:02:54.780130 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.780101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:02:54.791863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.791831    2578 generic.go:356] "Generic (PLEG): container finished" podID="66f01940-469b-4063-bce9-07fbe8661e8a" containerID="9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3" exitCode=2
Sep 09 21:02:54.791995 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.791879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerDied","Data":"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"}
Sep 09 21:02:54.793849 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.793828    2578 generic.go:356] "Generic (PLEG): container finished" podID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerID="cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb" exitCode=2
Sep 09 21:02:54.793970 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.793855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerDied","Data":"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"}
Sep 09 21:02:54.805443 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.805420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.805551 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.805486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgwn\" (UniqueName: \"kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.805632 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.805608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.906348 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.906310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgwn\" (UniqueName: \"kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.906506 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.906393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.906506 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.906430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.906615 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:54.906584    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-serving-cert: secret "success-200-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:54.906667 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:54.906651    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls podName:6a2ffa25-90c8-42fb-9de8-a996c393475e nodeName:}" failed. No retries permitted until 2026-09-09 21:02:55.40663173 +0000 UTC m=+622.571958990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls") pod "success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" (UID: "6a2ffa25-90c8-42fb-9de8-a996c393475e") : secret "success-200-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:54.907086 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.907062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.934553 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.934532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgwn\" (UniqueName: \"kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:54.935397 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.935377    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:02:54.937884 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.937867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:54.947484 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.947464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67ffc-predictor-serving-cert\""
Sep 09 21:02:54.951010 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.950995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67ffc-kube-rbac-proxy-sar-config\""
Sep 09 21:02:54.981270 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:54.981244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:02:55.007163 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.007101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.007289 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.007191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8kds\" (UniqueName: \"kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.007289 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.007216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.108372 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.108346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.108510 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.108411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8kds\" (UniqueName: \"kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.108569 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:55.108520    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-serving-cert: secret "error-404-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:55.108569 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.108538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.108670 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:55.108617    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls podName:d40d4be3-356c-4067-a949-16dee4d90ecb nodeName:}" failed. No retries permitted until 2026-09-09 21:02:55.608595018 +0000 UTC m=+622.773922284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls") pod "error-404-isvc-67ffc-predictor-d748749fb-2fpwx" (UID: "d40d4be3-356c-4067-a949-16dee4d90ecb") : secret "error-404-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:55.109322 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.109301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.123064 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.123043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8kds\" (UniqueName: \"kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.412146 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.412108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:55.412281 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:55.412246    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-serving-cert: secret "success-200-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:55.412333 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:55.412321    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls podName:6a2ffa25-90c8-42fb-9de8-a996c393475e nodeName:}" failed. No retries permitted until 2026-09-09 21:02:56.412301114 +0000 UTC m=+623.577628375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls") pod "success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" (UID: "6a2ffa25-90c8-42fb-9de8-a996c393475e") : secret "success-200-isvc-67ffc-predictor-serving-cert" not found
Sep 09 21:02:55.613388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.613356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.615805 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.615783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") pod \"error-404-isvc-67ffc-predictor-d748749fb-2fpwx\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:55.850193 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:55.850169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:56.134206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.134067    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:02:56.138133 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:02:56.138106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d4be3_356c_4067_a949_16dee4d90ecb.slice/crio-31eca5ef3a05f19a2ecb3c277e80637967b62c5fbd5460ba05097f3114bf19da WatchSource:0}: Error finding container 31eca5ef3a05f19a2ecb3c277e80637967b62c5fbd5460ba05097f3114bf19da: Status 404 returned error can't find the container with id 31eca5ef3a05f19a2ecb3c277e80637967b62c5fbd5460ba05097f3114bf19da
Sep 09 21:02:56.417375 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.417295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:56.419836 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.419819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") " pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:56.516317 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.516269    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 09 21:02:56.516471 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.516336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 09 21:02:56.565071 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.565048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:56.803720 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.803631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerStarted","Data":"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"}
Sep 09 21:02:56.803720 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.803702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerStarted","Data":"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"}
Sep 09 21:02:56.804126 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.803732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerStarted","Data":"31eca5ef3a05f19a2ecb3c277e80637967b62c5fbd5460ba05097f3114bf19da"}
Sep 09 21:02:56.804126 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.803870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:56.838344 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.838303    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podStartSLOduration=2.838287819 podStartE2EDuration="2.838287819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:56.835594003 +0000 UTC m=+624.000921271" watchObservedRunningTime="2026-09-09 21:02:56.838287819 +0000 UTC m=+624.003615087"
Sep 09 21:02:56.861770 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:56.861723    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:02:56.865636 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:02:56.865612    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2ffa25_90c8_42fb_9de8_a996c393475e.slice/crio-f3c0c4f9c66e0ef3ceaed8ea12802c22e21a7692908f3b4b2fb09ed7613ce4fa WatchSource:0}: Error finding container f3c0c4f9c66e0ef3ceaed8ea12802c22e21a7692908f3b4b2fb09ed7613ce4fa: Status 404 returned error can't find the container with id f3c0c4f9c66e0ef3ceaed8ea12802c22e21a7692908f3b4b2fb09ed7613ce4fa
Sep 09 21:02:57.706278 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.706258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:02:57.726704 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.726682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") pod \"66f01940-469b-4063-bce9-07fbe8661e8a\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") "
Sep 09 21:02:57.726855 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.726733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config\") pod \"66f01940-469b-4063-bce9-07fbe8661e8a\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") "
Sep 09 21:02:57.726855 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.726773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6khc\" (UniqueName: \"kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc\") pod \"66f01940-469b-4063-bce9-07fbe8661e8a\" (UID: \"66f01940-469b-4063-bce9-07fbe8661e8a\") "
Sep 09 21:02:57.727119 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.727097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66f01940-469b-4063-bce9-07fbe8661e8a" (UID: "66f01940-469b-4063-bce9-07fbe8661e8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:57.728904 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.728874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66f01940-469b-4063-bce9-07fbe8661e8a" (UID: "66f01940-469b-4063-bce9-07fbe8661e8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:57.729010 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.728985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc" (OuterVolumeSpecName: "kube-api-access-l6khc") pod "66f01940-469b-4063-bce9-07fbe8661e8a" (UID: "66f01940-469b-4063-bce9-07fbe8661e8a"). InnerVolumeSpecName "kube-api-access-l6khc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:57.738544 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.738501    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:02:57.809620 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.809588    2578 generic.go:356] "Generic (PLEG): container finished" podID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerID="6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79" exitCode=0
Sep 09 21:02:57.810018 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.809641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"
Sep 09 21:02:57.810018 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.809674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerDied","Data":"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"}
Sep 09 21:02:57.810018 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.809733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf" event={"ID":"03003a0f-4a04-4164-ab6a-f522ddbe9bf4","Type":"ContainerDied","Data":"234fdb84104de0161935e8fa8026e896329baadbe4a7a1daabb6132a5b2cd8a2"}
Sep 09 21:02:57.810018 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.809788    2578 scope.go:117] "RemoveContainer" containerID="cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"
Sep 09 21:02:57.811305 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.811286    2578 generic.go:356] "Generic (PLEG): container finished" podID="66f01940-469b-4063-bce9-07fbe8661e8a" containerID="7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d" exitCode=0
Sep 09 21:02:57.811404 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.811374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerDied","Data":"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"}
Sep 09 21:02:57.811446 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.811412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"
Sep 09 21:02:57.811446 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.811420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh" event={"ID":"66f01940-469b-4063-bce9-07fbe8661e8a","Type":"ContainerDied","Data":"3a8890fd62d2ce4e8e52712ed41a10f64705f06e45d89e7855f10248255ca80e"}
Sep 09 21:02:57.816334 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.816287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerStarted","Data":"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"}
Sep 09 21:02:57.816453 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.816342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerStarted","Data":"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"}
Sep 09 21:02:57.816453 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.816372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerStarted","Data":"f3c0c4f9c66e0ef3ceaed8ea12802c22e21a7692908f3b4b2fb09ed7613ce4fa"}
Sep 09 21:02:57.816858 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.816782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:02:57.817846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.817819    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:02:57.820729 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.820712    2578 scope.go:117] "RemoveContainer" containerID="6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"
Sep 09 21:02:57.827144 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config\") pod \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") "
Sep 09 21:02:57.827232 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkr8z\" (UniqueName: \"kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z\") pod \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") "
Sep 09 21:02:57.827232 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") pod \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\" (UID: \"03003a0f-4a04-4164-ab6a-f522ddbe9bf4\") "
Sep 09 21:02:57.827531 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03003a0f-4a04-4164-ab6a-f522ddbe9bf4" (UID: "03003a0f-4a04-4164-ab6a-f522ddbe9bf4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:57.827669 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827654    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f01940-469b-4063-bce9-07fbe8661e8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:57.827736 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6khc\" (UniqueName: \"kubernetes.io/projected/66f01940-469b-4063-bce9-07fbe8661e8a-kube-api-access-l6khc\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:57.827736 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827689    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:57.827736 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827704    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f01940-469b-4063-bce9-07fbe8661e8a-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:57.828026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.827993    2578 scope.go:117] "RemoveContainer" containerID="cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"
Sep 09 21:02:57.828394 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:57.828375    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb\": container with ID starting with cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb not found: ID does not exist" containerID="cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"
Sep 09 21:02:57.828467 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.828401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb"} err="failed to get container status \"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb\": rpc error: code = NotFound desc = could not find container \"cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb\": container with ID starting with cd5b063fcf433942691d9b37a2d48fb7933c449f3bc2a008f3529527a6e703eb not found: ID does not exist"
Sep 09 21:02:57.828467 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.828417    2578 scope.go:117] "RemoveContainer" containerID="6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"
Sep 09 21:02:57.828725 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:57.828706    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79\": container with ID starting with 6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79 not found: ID does not exist" containerID="6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"
Sep 09 21:02:57.828841 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.828728    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79"} err="failed to get container status \"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79\": rpc error: code = NotFound desc = could not find container \"6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79\": container with ID starting with 6ea110bc6b1034269607709d458a837d3657e3dce00e81d7d137ec8262c89a79 not found: ID does not exist"
Sep 09 21:02:57.828841 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.828791    2578 scope.go:117] "RemoveContainer" containerID="9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"
Sep 09 21:02:57.829960 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.829943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03003a0f-4a04-4164-ab6a-f522ddbe9bf4" (UID: "03003a0f-4a04-4164-ab6a-f522ddbe9bf4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:57.830229 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.830210    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z" (OuterVolumeSpecName: "kube-api-access-kkr8z") pod "03003a0f-4a04-4164-ab6a-f522ddbe9bf4" (UID: "03003a0f-4a04-4164-ab6a-f522ddbe9bf4"). InnerVolumeSpecName "kube-api-access-kkr8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:57.836205 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.836190    2578 scope.go:117] "RemoveContainer" containerID="7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"
Sep 09 21:02:57.843030 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.843008    2578 scope.go:117] "RemoveContainer" containerID="9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"
Sep 09 21:02:57.843319 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:57.843296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3\": container with ID starting with 9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3 not found: ID does not exist" containerID="9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"
Sep 09 21:02:57.843402 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.843324    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3"} err="failed to get container status \"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3\": rpc error: code = NotFound desc = could not find container \"9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3\": container with ID starting with 9139abca7b02d26c7753fc4c07238f26f5c5f9e9d7cd537a194b0bf0ca75f6f3 not found: ID does not exist"
Sep 09 21:02:57.843402 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.843339    2578 scope.go:117] "RemoveContainer" containerID="7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"
Sep 09 21:02:57.843578 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:02:57.843564    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d\": container with ID starting with 7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d not found: ID does not exist" containerID="7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"
Sep 09 21:02:57.843616 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.843580    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d"} err="failed to get container status \"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d\": rpc error: code = NotFound desc = could not find container \"7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d\": container with ID starting with 7008b1cf5166e82ed9f7efda46a4d093e61303d43d3a6e6d84d2718f006c170d not found: ID does not exist"
Sep 09 21:02:57.876414 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.876375    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podStartSLOduration=3.876359558 podStartE2EDuration="3.876359558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:57.875408123 +0000 UTC m=+625.040735390" watchObservedRunningTime="2026-09-09 21:02:57.876359558 +0000 UTC m=+625.041686829"
Sep 09 21:02:57.913250 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.913221    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:02:57.922957 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.922928    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-88167-predictor-df799f49b-g77jh"]
Sep 09 21:02:57.928044 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.928023    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkr8z\" (UniqueName: \"kubernetes.io/projected/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-kube-api-access-kkr8z\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:57.928106 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:57.928046    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03003a0f-4a04-4164-ab6a-f522ddbe9bf4-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:02:58.135388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.135358    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:02:58.141716 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.141697    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88167-predictor-57db869b7c-f4qxf"]
Sep 09 21:02:58.822424 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.822394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:58.822816 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.822455    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:02:58.822816 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.822467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:02:58.823575 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:58.823553    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:02:59.282397 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:59.282363    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" path="/var/lib/kubelet/pods/03003a0f-4a04-4164-ab6a-f522ddbe9bf4/volumes"
Sep 09 21:02:59.282789 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:59.282776    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" path="/var/lib/kubelet/pods/66f01940-469b-4063-bce9-07fbe8661e8a/volumes"
Sep 09 21:02:59.827022 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:02:59.826978    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:03:03.827050 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:03.827021    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:03:03.827508 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:03.827473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:03:04.831345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:04.831315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:03:04.831808 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:04.831777    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:03:13.828109 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:13.828057    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:03:14.832411 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:14.832359    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:03:23.828006 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:23.827967    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:03:24.832674 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:24.832629    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:03:33.827629 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:33.827589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 09 21:03:34.832690 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:34.832648    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 09 21:03:43.827940 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:43.827913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:03:44.402988 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.402955    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:03:44.403278 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403265    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403280    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403288    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kube-rbac-proxy"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403295    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kube-rbac-proxy"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403306    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403312    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container"
Sep 09 21:03:44.403320 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403319    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kube-rbac-proxy"
Sep 09 21:03:44.403492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403324    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kube-rbac-proxy"
Sep 09 21:03:44.403492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403370    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kserve-container"
Sep 09 21:03:44.403492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403378    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kube-rbac-proxy"
Sep 09 21:03:44.403492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403384    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03003a0f-4a04-4164-ab6a-f522ddbe9bf4" containerName="kserve-container"
Sep 09 21:03:44.403492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.403389    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f01940-469b-4063-bce9-07fbe8661e8a" containerName="kube-rbac-proxy"
Sep 09 21:03:44.406536 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.406521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.409143 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.409124    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-456dd-kube-rbac-proxy-sar-config\""
Sep 09 21:03:44.409282 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.409266    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-456dd-predictor-serving-cert\""
Sep 09 21:03:44.420866 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.420846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:03:44.443437 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.443413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.443547 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.443479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9q5\" (UniqueName: \"kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.443547 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.443502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.533591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.533563    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:03:44.535858 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.535843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.538432 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.538410    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-456dd-predictor-serving-cert\""
Sep 09 21:03:44.538729 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.538715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-456dd-kube-rbac-proxy-sar-config\""
Sep 09 21:03:44.544124 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.544104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.544243 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.544168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9q5\" (UniqueName: \"kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.544243 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.544197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.545066 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.545043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.546623 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.546606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.554170 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.554153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:03:44.559837 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.559820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9q5\" (UniqueName: \"kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5\") pod \"success-200-isvc-456dd-predictor-7948469779-dn998\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") " pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.644801 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.644777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.644900 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.644805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7d9h\" (UniqueName: \"kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.644900 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.644841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.716390 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.716332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:44.745789 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.745765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.745904 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.745798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7d9h\" (UniqueName: \"kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.745904 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.745823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.746476 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.746456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.748223 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.748203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.761088 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.761066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7d9h\" (UniqueName: \"kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h\") pod \"error-404-isvc-456dd-predictor-674f9c95c4-kdb9b\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") " pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.832571 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.832549    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:03:44.845684 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.845667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:44.993220 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:44.993199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:03:44.995044 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:03:44.995022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a8d3af_e895_44c6_baba_c357e05be25b.slice/crio-abaab1cd98ab3db160ee71e6d579740c4229aefc37a2de955a197b65f07d9de4 WatchSource:0}: Error finding container abaab1cd98ab3db160ee71e6d579740c4229aefc37a2de955a197b65f07d9de4: Status 404 returned error can't find the container with id abaab1cd98ab3db160ee71e6d579740c4229aefc37a2de955a197b65f07d9de4
Sep 09 21:03:45.018798 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:45.018775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerStarted","Data":"abaab1cd98ab3db160ee71e6d579740c4229aefc37a2de955a197b65f07d9de4"}
Sep 09 21:03:45.120793 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:45.119613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:03:46.024822 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.024787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerStarted","Data":"da5195c88ad928c69aa4268146155ee0c32b01865976289c8e67854813548592"}
Sep 09 21:03:46.025290 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.024850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerStarted","Data":"d3031c6bde83b56de6e784c4210e9928b1208477f94deddbb3ec94a15e43ccbd"}
Sep 09 21:03:46.025290 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.024940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:46.026782 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.026734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerStarted","Data":"0b6d6f3337a6fb990876109b4b911181cf814022972275ca7bc9fbe5a9a26ad6"}
Sep 09 21:03:46.026875 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.026815    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:46.026875 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.026848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:46.026937 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.026876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerStarted","Data":"04904ce3df2b605c0b51c40d603a3fdb0dd7f50c263810b10f93f614cafb0e48"}
Sep 09 21:03:46.026937 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.026905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerStarted","Data":"10bcfae95a0b9a5938abe1f6358dfc8bc5362922c9945a2f18058ad932b349ea"}
Sep 09 21:03:46.028020 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.027996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:03:46.083512 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.083461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podStartSLOduration=2.083447666 podStartE2EDuration="2.083447666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:46.082545382 +0000 UTC m=+673.247872657" watchObservedRunningTime="2026-09-09 21:03:46.083447666 +0000 UTC m=+673.248774933"
Sep 09 21:03:46.083647 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:46.083611    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podStartSLOduration=2.083606652 podStartE2EDuration="2.083606652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:46.058610047 +0000 UTC m=+673.223937316" watchObservedRunningTime="2026-09-09 21:03:46.083606652 +0000 UTC m=+673.248933919"
Sep 09 21:03:47.031665 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:47.031627    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:03:47.032123 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:47.031782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:47.032902 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:47.032879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:03:48.035589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:48.035553    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:03:52.037199 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:52.037164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:03:52.037583 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:52.037563    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:03:53.039819 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:53.039790    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:03:53.040421 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:03:53.040391    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:04:02.037811 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:02.037774    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:04:03.041152 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:03.041110    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:04:12.037812 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:12.037772    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:04:13.040800 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:13.040764    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:04:22.037913 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:22.037874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:04:23.041168 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:23.041125    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 09 21:04:32.038561 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:32.038482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:04:33.040898 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:04:33.040873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:12:19.381679 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.381638    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:12:19.382300 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.381894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container" containerID="cri-o://765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919" gracePeriod=30
Sep 09 21:12:19.382300 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.382009    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kube-rbac-proxy" containerID="cri-o://d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0" gracePeriod=30
Sep 09 21:12:19.520252 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.520223    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:12:19.522556 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.522537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.527860 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.527842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-71ac5-predictor-serving-cert\""
Sep 09 21:12:19.528252 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.527847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-71ac5-kube-rbac-proxy-sar-config\""
Sep 09 21:12:19.531870 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.531848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:12:19.532111 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.532065    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container" containerID="cri-o://1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7" gracePeriod=30
Sep 09 21:12:19.532192 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.532129    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kube-rbac-proxy" containerID="cri-o://937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a" gracePeriod=30
Sep 09 21:12:19.543077 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.542950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:12:19.600510 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.600480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.600670 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.600516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.600670 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.600542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7hrm\" (UniqueName: \"kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.603991 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.603969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:12:19.606268 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.606252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.610572 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.610553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-71ac5-kube-rbac-proxy-sar-config\""
Sep 09 21:12:19.610882 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.610866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-71ac5-predictor-serving-cert\""
Sep 09 21:12:19.622429 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.622410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:12:19.701025 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.700990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.701162 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.701162 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.701162 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8llv\" (UniqueName: \"kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.701162 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hrm\" (UniqueName: \"kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.701364 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.701671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.701654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.703488 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.703460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.723145 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.723123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hrm\" (UniqueName: \"kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm\") pod \"success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:19.802322 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.802297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.802432 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.802331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.802432 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.802367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8llv\" (UniqueName: \"kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.802547 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:19.802474    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-serving-cert: secret "error-404-isvc-71ac5-predictor-serving-cert" not found
Sep 09 21:12:19.802602 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:19.802553    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls podName:b5fab6ce-1fa3-43a9-8345-04945b21d047 nodeName:}" failed. No retries permitted until 2026-09-09 21:12:20.302532834 +0000 UTC m=+1187.467860083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls") pod "error-404-isvc-71ac5-predictor-7775796c74-h95dv" (UID: "b5fab6ce-1fa3-43a9-8345-04945b21d047") : secret "error-404-isvc-71ac5-predictor-serving-cert" not found
Sep 09 21:12:19.802911 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.802895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.812048 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.812018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8llv\" (UniqueName: \"kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:19.827990 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.827964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 09 21:12:19.836238 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:19.836220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:20.124684 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.124658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:12:20.126960 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:12:20.126936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb874311_ea63_403c_b05f_c0668bfbbf1f.slice/crio-b2118d0dfa4f8b75b461f90fbefe25443134d8a9df8bacf75279857a70574568 WatchSource:0}: Error finding container b2118d0dfa4f8b75b461f90fbefe25443134d8a9df8bacf75279857a70574568: Status 404 returned error can't find the container with id b2118d0dfa4f8b75b461f90fbefe25443134d8a9df8bacf75279857a70574568
Sep 09 21:12:20.136205 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.136183    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerID="d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0" exitCode=2
Sep 09 21:12:20.136296 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.136248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerDied","Data":"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"}
Sep 09 21:12:20.137389 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.137364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerStarted","Data":"b2118d0dfa4f8b75b461f90fbefe25443134d8a9df8bacf75279857a70574568"}
Sep 09 21:12:20.139112 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.139087    2578 generic.go:356] "Generic (PLEG): container finished" podID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerID="937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a" exitCode=2
Sep 09 21:12:20.139189 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.139155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerDied","Data":"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"}
Sep 09 21:12:20.306120 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.306090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:20.308428 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.308412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") pod \"error-404-isvc-71ac5-predictor-7775796c74-h95dv\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:20.516424 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.516348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:20.791118 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:20.791093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:12:20.793423 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:12:20.793392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fab6ce_1fa3_43a9_8345_04945b21d047.slice/crio-f0e40d813717bda3ef653a80bf82b3671f2f4cc79a04e702d240f4f6607de065 WatchSource:0}: Error finding container f0e40d813717bda3ef653a80bf82b3671f2f4cc79a04e702d240f4f6607de065: Status 404 returned error can't find the container with id f0e40d813717bda3ef653a80bf82b3671f2f4cc79a04e702d240f4f6607de065
Sep 09 21:12:21.146508 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.146469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerStarted","Data":"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"}
Sep 09 21:12:21.146666 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.146644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:21.146785 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.146685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:21.146785 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.146711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerStarted","Data":"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"}
Sep 09 21:12:21.146785 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.146761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerStarted","Data":"f0e40d813717bda3ef653a80bf82b3671f2f4cc79a04e702d240f4f6607de065"}
Sep 09 21:12:21.148083 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.148056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:21.148388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.148368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerStarted","Data":"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"}
Sep 09 21:12:21.148481 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.148411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerStarted","Data":"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"}
Sep 09 21:12:21.148538 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.148519    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:21.175609 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.175574    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podStartSLOduration=2.175558223 podStartE2EDuration="2.175558223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:21.17500176 +0000 UTC m=+1188.340329028" watchObservedRunningTime="2026-09-09 21:12:21.175558223 +0000 UTC m=+1188.340885491"
Sep 09 21:12:21.206195 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:21.206151    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podStartSLOduration=2.206137372 podStartE2EDuration="2.206137372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:21.203652337 +0000 UTC m=+1188.368979604" watchObservedRunningTime="2026-09-09 21:12:21.206137372 +0000 UTC m=+1188.371464639"
Sep 09 21:12:22.153292 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.153253    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:22.153709 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.153267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:22.154321 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.154297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:22.570911 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.570891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:12:22.623945 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.623916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") pod \"d40d4be3-356c-4067-a949-16dee4d90ecb\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") "
Sep 09 21:12:22.624071 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.623968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config\") pod \"d40d4be3-356c-4067-a949-16dee4d90ecb\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") "
Sep 09 21:12:22.624071 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.623992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8kds\" (UniqueName: \"kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds\") pod \"d40d4be3-356c-4067-a949-16dee4d90ecb\" (UID: \"d40d4be3-356c-4067-a949-16dee4d90ecb\") "
Sep 09 21:12:22.624309 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.624286    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d40d4be3-356c-4067-a949-16dee4d90ecb" (UID: "d40d4be3-356c-4067-a949-16dee4d90ecb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:22.626082 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.626061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d40d4be3-356c-4067-a949-16dee4d90ecb" (UID: "d40d4be3-356c-4067-a949-16dee4d90ecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:22.626154 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.626102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds" (OuterVolumeSpecName: "kube-api-access-l8kds") pod "d40d4be3-356c-4067-a949-16dee4d90ecb" (UID: "d40d4be3-356c-4067-a949-16dee4d90ecb"). InnerVolumeSpecName "kube-api-access-l8kds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:22.725519 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.725457    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:22.725519 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.725490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8kds\" (UniqueName: \"kubernetes.io/projected/d40d4be3-356c-4067-a949-16dee4d90ecb-kube-api-access-l8kds\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:22.725519 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:22.725505    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d40d4be3-356c-4067-a949-16dee4d90ecb-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:23.014968 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.014940    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:12:23.027290 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.027265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") pod \"6a2ffa25-90c8-42fb-9de8-a996c393475e\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") "
Sep 09 21:12:23.027397 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.027304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config\") pod \"6a2ffa25-90c8-42fb-9de8-a996c393475e\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") "
Sep 09 21:12:23.027678 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.027657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a2ffa25-90c8-42fb-9de8-a996c393475e" (UID: "6a2ffa25-90c8-42fb-9de8-a996c393475e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:23.029352 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.029335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a2ffa25-90c8-42fb-9de8-a996c393475e" (UID: "6a2ffa25-90c8-42fb-9de8-a996c393475e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:23.128324 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.128299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcgwn\" (UniqueName: \"kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn\") pod \"6a2ffa25-90c8-42fb-9de8-a996c393475e\" (UID: \"6a2ffa25-90c8-42fb-9de8-a996c393475e\") "
Sep 09 21:12:23.128468 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.128406    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a2ffa25-90c8-42fb-9de8-a996c393475e-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:23.128468 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.128416    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:23.130311 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.130291    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn" (OuterVolumeSpecName: "kube-api-access-fcgwn") pod "6a2ffa25-90c8-42fb-9de8-a996c393475e" (UID: "6a2ffa25-90c8-42fb-9de8-a996c393475e"). InnerVolumeSpecName "kube-api-access-fcgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:23.159225 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.159200    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerID="765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919" exitCode=0
Sep 09 21:12:23.159572 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.159251    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"
Sep 09 21:12:23.159572 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.159277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerDied","Data":"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"}
Sep 09 21:12:23.159572 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.159330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm" event={"ID":"6a2ffa25-90c8-42fb-9de8-a996c393475e","Type":"ContainerDied","Data":"f3c0c4f9c66e0ef3ceaed8ea12802c22e21a7692908f3b4b2fb09ed7613ce4fa"}
Sep 09 21:12:23.159572 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.159368    2578 scope.go:117] "RemoveContainer" containerID="d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"
Sep 09 21:12:23.161117 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.161094    2578 generic.go:356] "Generic (PLEG): container finished" podID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerID="1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7" exitCode=0
Sep 09 21:12:23.161217 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.161138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerDied","Data":"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"}
Sep 09 21:12:23.161217 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.161155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"
Sep 09 21:12:23.161217 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.161187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx" event={"ID":"d40d4be3-356c-4067-a949-16dee4d90ecb","Type":"ContainerDied","Data":"31eca5ef3a05f19a2ecb3c277e80637967b62c5fbd5460ba05097f3114bf19da"}
Sep 09 21:12:23.161650 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.161626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:23.167687 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.167672    2578 scope.go:117] "RemoveContainer" containerID="765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"
Sep 09 21:12:23.175256 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.175237    2578 scope.go:117] "RemoveContainer" containerID="d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"
Sep 09 21:12:23.175522 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:23.175505    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0\": container with ID starting with d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0 not found: ID does not exist" containerID="d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"
Sep 09 21:12:23.175574 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.175529    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0"} err="failed to get container status \"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0\": rpc error: code = NotFound desc = could not find container \"d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0\": container with ID starting with d1ea5224e1a42221f84cfec0263c41f5cc5131a874211bd5668a29ad62b590e0 not found: ID does not exist"
Sep 09 21:12:23.175574 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.175545    2578 scope.go:117] "RemoveContainer" containerID="765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"
Sep 09 21:12:23.175817 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:23.175799    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919\": container with ID starting with 765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919 not found: ID does not exist" containerID="765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"
Sep 09 21:12:23.175878 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.175822    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919"} err="failed to get container status \"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919\": rpc error: code = NotFound desc = could not find container \"765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919\": container with ID starting with 765ffd62bef5a58d35d972e6e2a70d2d90b79ecf0ab3f864a77ec76b8d4c8919 not found: ID does not exist"
Sep 09 21:12:23.175878 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.175836    2578 scope.go:117] "RemoveContainer" containerID="937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"
Sep 09 21:12:23.183868 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.183807    2578 scope.go:117] "RemoveContainer" containerID="1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"
Sep 09 21:12:23.186043 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.186022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:12:23.191854 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.191838    2578 scope.go:117] "RemoveContainer" containerID="937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"
Sep 09 21:12:23.192120 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:23.192103    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a\": container with ID starting with 937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a not found: ID does not exist" containerID="937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"
Sep 09 21:12:23.192176 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.192132    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a"} err="failed to get container status \"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a\": rpc error: code = NotFound desc = could not find container \"937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a\": container with ID starting with 937fd71c31dc422b40213502f88c9d7885a1f356b7ce5a53e7888547fc4b218a not found: ID does not exist"
Sep 09 21:12:23.192176 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.192148    2578 scope.go:117] "RemoveContainer" containerID="1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"
Sep 09 21:12:23.192373 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:23.192355    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7\": container with ID starting with 1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7 not found: ID does not exist" containerID="1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"
Sep 09 21:12:23.192426 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.192382    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7"} err="failed to get container status \"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7\": rpc error: code = NotFound desc = could not find container \"1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7\": container with ID starting with 1f026d90c6c8ebafd84f83349cb0f8f519b61377a3c383212a540ab86d18d5b7 not found: ID does not exist"
Sep 09 21:12:23.192873 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.192857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ffc-predictor-d748749fb-2fpwx"]
Sep 09 21:12:23.204762 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.204726    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:12:23.210803 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.210786    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ffc-predictor-6f9b4c645c-kfrkm"]
Sep 09 21:12:23.229147 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.229127    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgwn\" (UniqueName: \"kubernetes.io/projected/6a2ffa25-90c8-42fb-9de8-a996c393475e-kube-api-access-fcgwn\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:12:23.283293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.283226    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" path="/var/lib/kubelet/pods/6a2ffa25-90c8-42fb-9de8-a996c393475e/volumes"
Sep 09 21:12:23.283697 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:23.283679    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" path="/var/lib/kubelet/pods/d40d4be3-356c-4067-a949-16dee4d90ecb/volumes"
Sep 09 21:12:27.157692 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:27.157666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:12:27.158181 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:27.158141    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:28.166228 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:28.166200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:12:28.166668 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:28.166644    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:37.158315 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:37.158271    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:38.167459 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:38.167418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:47.159044 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:47.159003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:48.166822 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:48.166784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:57.158201 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:57.158159    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:12:58.166798 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:58.166763    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 09 21:12:58.991259 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:58.991229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:12:58.991481 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:58.991454    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container" containerID="cri-o://d3031c6bde83b56de6e784c4210e9928b1208477f94deddbb3ec94a15e43ccbd" gracePeriod=30
Sep 09 21:12:58.991543 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:58.991496    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kube-rbac-proxy" containerID="cri-o://da5195c88ad928c69aa4268146155ee0c32b01865976289c8e67854813548592" gracePeriod=30
Sep 09 21:12:59.055418 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.055390    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:12:59.055652 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.055624    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kube-rbac-proxy" containerID="cri-o://0b6d6f3337a6fb990876109b4b911181cf814022972275ca7bc9fbe5a9a26ad6" gracePeriod=30
Sep 09 21:12:59.055776 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.055614    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" containerID="cri-o://04904ce3df2b605c0b51c40d603a3fdb0dd7f50c263810b10f93f614cafb0e48" gracePeriod=30
Sep 09 21:12:59.173769 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.173730    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:12:59.174095 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174082    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174098    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174112    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174119    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container"
Sep 09 21:12:59.174142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174132    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174144    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174150    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174214    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kube-rbac-proxy"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174229    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40d4be3-356c-4067-a949-16dee4d90ecb" containerName="kserve-container"
Sep 09 21:12:59.174293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.174235    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2ffa25-90c8-42fb-9de8-a996c393475e" containerName="kserve-container"
Sep 09 21:12:59.176216 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.176202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.178764 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.178719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07fe8-predictor-serving-cert\""
Sep 09 21:12:59.179063 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.178813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07fe8-kube-rbac-proxy-sar-config\""
Sep 09 21:12:59.198774 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.198733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:12:59.292996 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.292922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.292996 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.292959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqgj\" (UniqueName: \"kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.293164 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.293040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.315234 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.315211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:12:59.317661 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.317647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.321917 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.321897    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-07fe8-predictor-serving-cert\""
Sep 09 21:12:59.322013 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.321900    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-07fe8-kube-rbac-proxy-sar-config\""
Sep 09 21:12:59.326247 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.326227    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerID="da5195c88ad928c69aa4268146155ee0c32b01865976289c8e67854813548592" exitCode=2
Sep 09 21:12:59.326331 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.326272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerDied","Data":"da5195c88ad928c69aa4268146155ee0c32b01865976289c8e67854813548592"}
Sep 09 21:12:59.328136 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.328115    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerID="0b6d6f3337a6fb990876109b4b911181cf814022972275ca7bc9fbe5a9a26ad6" exitCode=2
Sep 09 21:12:59.328229 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.328155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerDied","Data":"0b6d6f3337a6fb990876109b4b911181cf814022972275ca7bc9fbe5a9a26ad6"}
Sep 09 21:12:59.332137 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.332120    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:12:59.393807 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.393780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqgj\" (UniqueName: \"kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.393934 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.393818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.393934 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.393871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.394021 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.393940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.394021 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.393989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.394124 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.394028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.394587 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.394565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.396571 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.396550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.407287 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.407260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqgj\" (UniqueName: \"kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj\") pod \"success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") " pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.488248 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.488218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:12:59.495312 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.495290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.495405 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.495339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.495405 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.495376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.495517 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:59.495436    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-serving-cert: secret "error-404-isvc-07fe8-predictor-serving-cert" not found
Sep 09 21:12:59.495517 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:12:59.495506    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls podName:8fd0fce0-66cc-4939-9d3c-27ac27674902 nodeName:}" failed. No retries permitted until 2026-09-09 21:12:59.995490175 +0000 UTC m=+1227.160817421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls") pod "error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" (UID: "8fd0fce0-66cc-4939-9d3c-27ac27674902") : secret "error-404-isvc-07fe8-predictor-serving-cert" not found
Sep 09 21:12:59.496026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.496009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.512461 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.512425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:12:59.773895 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:12:59.773863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:12:59.777471 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:12:59.777442    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c4adf6_9ca1_494d_a53a_588ece6f8d98.slice/crio-cd54760ad8261643582028623588b139498e8cbf5e0572f886d0c57debce5a61 WatchSource:0}: Error finding container cd54760ad8261643582028623588b139498e8cbf5e0572f886d0c57debce5a61: Status 404 returned error can't find the container with id cd54760ad8261643582028623588b139498e8cbf5e0572f886d0c57debce5a61
Sep 09 21:13:00.000598 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.000564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:00.002933 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.002912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") pod \"error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") " pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:00.228941 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.228912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:00.334781 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.334738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerStarted","Data":"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"}
Sep 09 21:13:00.334922 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.334818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerStarted","Data":"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"}
Sep 09 21:13:00.334922 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.334850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerStarted","Data":"cd54760ad8261643582028623588b139498e8cbf5e0572f886d0c57debce5a61"}
Sep 09 21:13:00.334922 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.334886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:13:00.357334 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.357290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podStartSLOduration=1.357274012 podStartE2EDuration="1.357274012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:00.356882925 +0000 UTC m=+1227.522210195" watchObservedRunningTime="2026-09-09 21:13:00.357274012 +0000 UTC m=+1227.522601280"
Sep 09 21:13:00.513696 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:00.513673    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:13:00.515302 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:13:00.515272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd0fce0_66cc_4939_9d3c_27ac27674902.slice/crio-a4e52614c7b2d5d2e102ad6ac14988b6b2644cea900c50875ae5c87ff7a77a2a WatchSource:0}: Error finding container a4e52614c7b2d5d2e102ad6ac14988b6b2644cea900c50875ae5c87ff7a77a2a: Status 404 returned error can't find the container with id a4e52614c7b2d5d2e102ad6ac14988b6b2644cea900c50875ae5c87ff7a77a2a
Sep 09 21:13:01.340959 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.340919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerStarted","Data":"afaafaa7677d61524c9b4782282c0b034c06bf838c8fa28b2448b93f9fbcd5fa"}
Sep 09 21:13:01.341433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.340980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerStarted","Data":"7cdf2aa47f213f1c8b1643b600427d8048eb2d96fb60b16b4992ca33e631ae5e"}
Sep 09 21:13:01.341433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.341009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerStarted","Data":"a4e52614c7b2d5d2e102ad6ac14988b6b2644cea900c50875ae5c87ff7a77a2a"}
Sep 09 21:13:01.341433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.341200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:01.341433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.341243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:01.341433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.341264    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:13:01.342231 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.342206    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:01.342316 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.342210    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:01.372801 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:01.372762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podStartSLOduration=2.37273047 podStartE2EDuration="2.37273047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:01.3721535 +0000 UTC m=+1228.537480817" watchObservedRunningTime="2026-09-09 21:13:01.37273047 +0000 UTC m=+1228.538057733"
Sep 09 21:13:02.032350 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.032310    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 09 21:13:02.037761 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.037717    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 09 21:13:02.348674 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.348633    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerID="d3031c6bde83b56de6e784c4210e9928b1208477f94deddbb3ec94a15e43ccbd" exitCode=0
Sep 09 21:13:02.349096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.348684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerDied","Data":"d3031c6bde83b56de6e784c4210e9928b1208477f94deddbb3ec94a15e43ccbd"}
Sep 09 21:13:02.349096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.349040    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:02.349202 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.349153    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:02.438488 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.438464    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:13:02.521683 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.521657    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls\") pod \"c6a8d3af-e895-44c6-baba-c357e05be25b\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") "
Sep 09 21:13:02.521844 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.521730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9q5\" (UniqueName: \"kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5\") pod \"c6a8d3af-e895-44c6-baba-c357e05be25b\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") "
Sep 09 21:13:02.521844 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.521768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config\") pod \"c6a8d3af-e895-44c6-baba-c357e05be25b\" (UID: \"c6a8d3af-e895-44c6-baba-c357e05be25b\") "
Sep 09 21:13:02.522133 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.522099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6a8d3af-e895-44c6-baba-c357e05be25b" (UID: "c6a8d3af-e895-44c6-baba-c357e05be25b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:02.523677 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.523655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6a8d3af-e895-44c6-baba-c357e05be25b" (UID: "c6a8d3af-e895-44c6-baba-c357e05be25b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:02.523809 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.523789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5" (OuterVolumeSpecName: "kube-api-access-dd9q5") pod "c6a8d3af-e895-44c6-baba-c357e05be25b" (UID: "c6a8d3af-e895-44c6-baba-c357e05be25b"). InnerVolumeSpecName "kube-api-access-dd9q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:02.623145 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.623108    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a8d3af-e895-44c6-baba-c357e05be25b-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:02.623145 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.623136    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9q5\" (UniqueName: \"kubernetes.io/projected/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-api-access-dd9q5\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:02.623145 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:02.623151    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a8d3af-e895-44c6-baba-c357e05be25b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:03.354906 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.354882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"
Sep 09 21:13:03.355348 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.354878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998" event={"ID":"c6a8d3af-e895-44c6-baba-c357e05be25b","Type":"ContainerDied","Data":"abaab1cd98ab3db160ee71e6d579740c4229aefc37a2de955a197b65f07d9de4"}
Sep 09 21:13:03.355348 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.355051    2578 scope.go:117] "RemoveContainer" containerID="da5195c88ad928c69aa4268146155ee0c32b01865976289c8e67854813548592"
Sep 09 21:13:03.357310 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.357285    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerID="04904ce3df2b605c0b51c40d603a3fdb0dd7f50c263810b10f93f614cafb0e48" exitCode=0
Sep 09 21:13:03.357562 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.357384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerDied","Data":"04904ce3df2b605c0b51c40d603a3fdb0dd7f50c263810b10f93f614cafb0e48"}
Sep 09 21:13:03.365694 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.365674    2578 scope.go:117] "RemoveContainer" containerID="d3031c6bde83b56de6e784c4210e9928b1208477f94deddbb3ec94a15e43ccbd"
Sep 09 21:13:03.378212 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.378187    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:13:03.383790 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.383766    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-456dd-predictor-7948469779-dn998"]
Sep 09 21:13:03.407052 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.407034    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:13:03.529964 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.529900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7d9h\" (UniqueName: \"kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h\") pod \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") "
Sep 09 21:13:03.529964 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.529936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls\") pod \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") "
Sep 09 21:13:03.530113 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.530006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config\") pod \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\" (UID: \"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d\") "
Sep 09 21:13:03.530376 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.530355    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" (UID: "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:03.532013 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.531991    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h" (OuterVolumeSpecName: "kube-api-access-z7d9h") pod "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" (UID: "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d"). InnerVolumeSpecName "kube-api-access-z7d9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:03.532196 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.532180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" (UID: "fa516c96-4bf2-4c13-bbd2-0142a16cbe8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:03.631292 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.631263    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7d9h\" (UniqueName: \"kubernetes.io/projected/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-api-access-z7d9h\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:03.631292 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.631287    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:03.631292 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:03.631296    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:04.364311 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.364281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b" event={"ID":"fa516c96-4bf2-4c13-bbd2-0142a16cbe8d","Type":"ContainerDied","Data":"10bcfae95a0b9a5938abe1f6358dfc8bc5362922c9945a2f18058ad932b349ea"}
Sep 09 21:13:04.364724 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.364342    2578 scope.go:117] "RemoveContainer" containerID="0b6d6f3337a6fb990876109b4b911181cf814022972275ca7bc9fbe5a9a26ad6"
Sep 09 21:13:04.364724 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.364291    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"
Sep 09 21:13:04.373042 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.373024    2578 scope.go:117] "RemoveContainer" containerID="04904ce3df2b605c0b51c40d603a3fdb0dd7f50c263810b10f93f614cafb0e48"
Sep 09 21:13:04.396796 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.396772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:13:04.403897 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:04.403876    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-456dd-predictor-674f9c95c4-kdb9b"]
Sep 09 21:13:05.283438 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:05.283406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" path="/var/lib/kubelet/pods/c6a8d3af-e895-44c6-baba-c357e05be25b/volumes"
Sep 09 21:13:05.283818 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:05.283805    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" path="/var/lib/kubelet/pods/fa516c96-4bf2-4c13-bbd2-0142a16cbe8d/volumes"
Sep 09 21:13:07.158697 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:07.158661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:13:07.353509 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:07.353480    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:13:07.353906 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:07.353887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:07.353987 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:07.353950    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:07.354253 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:07.354229    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:08.167719 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:08.167685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:13:17.354731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:17.354697    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:17.355107 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:17.354697    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:27.353942 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:27.353851    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:27.354359 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:27.354278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:30.066215 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.066182    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:13:30.066684 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.066446    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container" containerID="cri-o://93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463" gracePeriod=30
Sep 09 21:13:30.066684 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.066466    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kube-rbac-proxy" containerID="cri-o://444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c" gracePeriod=30
Sep 09 21:13:30.277636 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.277602    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:13:30.278119 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278099    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container"
Sep 09 21:13:30.278168 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278125    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container"
Sep 09 21:13:30.278168 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278153    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kube-rbac-proxy"
Sep 09 21:13:30.278168 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278161    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kube-rbac-proxy"
Sep 09 21:13:30.278280 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278169    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container"
Sep 09 21:13:30.278280 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278178    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container"
Sep 09 21:13:30.278280 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278192    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kube-rbac-proxy"
Sep 09 21:13:30.278280 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278201    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kube-rbac-proxy"
Sep 09 21:13:30.278396 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kserve-container"
Sep 09 21:13:30.278396 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278301    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kserve-container"
Sep 09 21:13:30.278396 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278312    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa516c96-4bf2-4c13-bbd2-0142a16cbe8d" containerName="kube-rbac-proxy"
Sep 09 21:13:30.278396 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.278321    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a8d3af-e895-44c6-baba-c357e05be25b" containerName="kube-rbac-proxy"
Sep 09 21:13:30.281798 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.281775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.288240 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.288216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e5cb-predictor-serving-cert\""
Sep 09 21:13:30.288354 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.288222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e5cb-kube-rbac-proxy-sar-config\""
Sep 09 21:13:30.300401 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.300378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:13:30.345978 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.345952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.346096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.346007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.346096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.346043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4k9\" (UniqueName: \"kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.418260 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.418218    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:13:30.418514 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.418486    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" containerID="cri-o://f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea" gracePeriod=30
Sep 09 21:13:30.418617 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.418544    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy" containerID="cri-o://242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744" gracePeriod=30
Sep 09 21:13:30.446630 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.446603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.446725 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.446659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.446725 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.446680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4k9\" (UniqueName: \"kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.446856 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:30.446789    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-serving-cert: secret "success-200-isvc-2e5cb-predictor-serving-cert" not found
Sep 09 21:13:30.446902 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:30.446856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls podName:ad98443a-63a8-45d9-8e57-e23ba7cc1779 nodeName:}" failed. No retries permitted until 2026-09-09 21:13:30.946836508 +0000 UTC m=+1258.112163771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls") pod "success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" (UID: "ad98443a-63a8-45d9-8e57-e23ba7cc1779") : secret "success-200-isvc-2e5cb-predictor-serving-cert" not found
Sep 09 21:13:30.447243 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.447224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.467739 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.467709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4k9\" (UniqueName: \"kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.479671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.479632    2578 generic.go:356] "Generic (PLEG): container finished" podID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerID="444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c" exitCode=2
Sep 09 21:13:30.479792 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.479701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerDied","Data":"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"}
Sep 09 21:13:30.720014 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.719942    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:13:30.723201 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.723183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.731730 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.731697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e5cb-predictor-serving-cert\""
Sep 09 21:13:30.731730 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.731711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e5cb-kube-rbac-proxy-sar-config\""
Sep 09 21:13:30.749147 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.749115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.749253 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.749229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.749318 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.749271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.769165 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.769143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:13:30.849654 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.849624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.849839 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.849660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.849839 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.849698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.849839 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:30.849810    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-serving-cert: secret "error-404-isvc-2e5cb-predictor-serving-cert" not found
Sep 09 21:13:30.849981 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:30.849878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls podName:5694902d-0f73-4a27-9a32-855abbf57462 nodeName:}" failed. No retries permitted until 2026-09-09 21:13:31.349863383 +0000 UTC m=+1258.515190629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls") pod "error-404-isvc-2e5cb-predictor-bcc98846-sthd8" (UID: "5694902d-0f73-4a27-9a32-855abbf57462") : secret "error-404-isvc-2e5cb-predictor-serving-cert" not found
Sep 09 21:13:30.850346 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.850327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.884591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.884562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:30.950060 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.950029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:30.952427 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:30.952408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") pod \"success-200-isvc-2e5cb-predictor-5766cb6459-k46x6\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") " pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:31.193579 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.193547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:31.353114 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.353082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:31.355463 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.355437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") pod \"error-404-isvc-2e5cb-predictor-bcc98846-sthd8\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") " pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:31.479386 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.479351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:13:31.480724 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:13:31.480694    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad98443a_63a8_45d9_8e57_e23ba7cc1779.slice/crio-6aeae0874b8230d20ffddff69e24bbe4b9a5b02ca3937b0e5190f71d08584774 WatchSource:0}: Error finding container 6aeae0874b8230d20ffddff69e24bbe4b9a5b02ca3937b0e5190f71d08584774: Status 404 returned error can't find the container with id 6aeae0874b8230d20ffddff69e24bbe4b9a5b02ca3937b0e5190f71d08584774
Sep 09 21:13:31.487832 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.487807    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerID="242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744" exitCode=2
Sep 09 21:13:31.487912 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.487874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerDied","Data":"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"}
Sep 09 21:13:31.634115 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.634086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:31.919095 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:31.919072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:13:31.921077 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:13:31.921054    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5694902d_0f73_4a27_9a32_855abbf57462.slice/crio-9c7a1095de6c4fdbfc6584325cfb2d910dca0c0c9e9043786b5e361002fe7a66 WatchSource:0}: Error finding container 9c7a1095de6c4fdbfc6584325cfb2d910dca0c0c9e9043786b5e361002fe7a66: Status 404 returned error can't find the container with id 9c7a1095de6c4fdbfc6584325cfb2d910dca0c0c9e9043786b5e361002fe7a66
Sep 09 21:13:32.153643 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.153605    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 09 21:13:32.498877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.498831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerStarted","Data":"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"}
Sep 09 21:13:32.499345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.498936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerStarted","Data":"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"}
Sep 09 21:13:32.499345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.498972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:32.499345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.498999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerStarted","Data":"6aeae0874b8230d20ffddff69e24bbe4b9a5b02ca3937b0e5190f71d08584774"}
Sep 09 21:13:32.499345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.499096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:32.500491 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.500464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:13:32.501250 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.501226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerStarted","Data":"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"}
Sep 09 21:13:32.501365 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.501278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerStarted","Data":"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"}
Sep 09 21:13:32.501365 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.501307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerStarted","Data":"9c7a1095de6c4fdbfc6584325cfb2d910dca0c0c9e9043786b5e361002fe7a66"}
Sep 09 21:13:32.501365 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.501340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:32.501497 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.501373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:32.502372 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.502353    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:13:32.523731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.523689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podStartSLOduration=2.523678632 podStartE2EDuration="2.523678632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:32.522498487 +0000 UTC m=+1259.687825755" watchObservedRunningTime="2026-09-09 21:13:32.523678632 +0000 UTC m=+1259.689005900"
Sep 09 21:13:32.546384 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:32.546340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podStartSLOduration=2.54632747 podStartE2EDuration="2.54632747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:32.544605128 +0000 UTC m=+1259.709932417" watchObservedRunningTime="2026-09-09 21:13:32.54632747 +0000 UTC m=+1259.711654749"
Sep 09 21:13:33.162530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.162493    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 09 21:13:33.317877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.317854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:13:33.365769 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.365688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config\") pod \"db874311-ea63-403c-b05f-c0668bfbbf1f\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") "
Sep 09 21:13:33.365769 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.365728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls\") pod \"db874311-ea63-403c-b05f-c0668bfbbf1f\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") "
Sep 09 21:13:33.365961 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.365775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7hrm\" (UniqueName: \"kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm\") pod \"db874311-ea63-403c-b05f-c0668bfbbf1f\" (UID: \"db874311-ea63-403c-b05f-c0668bfbbf1f\") "
Sep 09 21:13:33.366050 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.366026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db874311-ea63-403c-b05f-c0668bfbbf1f" (UID: "db874311-ea63-403c-b05f-c0668bfbbf1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:33.367960 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.367932    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db874311-ea63-403c-b05f-c0668bfbbf1f" (UID: "db874311-ea63-403c-b05f-c0668bfbbf1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:33.368045 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.367998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm" (OuterVolumeSpecName: "kube-api-access-j7hrm") pod "db874311-ea63-403c-b05f-c0668bfbbf1f" (UID: "db874311-ea63-403c-b05f-c0668bfbbf1f"). InnerVolumeSpecName "kube-api-access-j7hrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:33.466589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.466559    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db874311-ea63-403c-b05f-c0668bfbbf1f-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:33.466589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.466580    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7hrm\" (UniqueName: \"kubernetes.io/projected/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-api-access-j7hrm\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:33.466589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.466589    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db874311-ea63-403c-b05f-c0668bfbbf1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:33.509105 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509074    2578 generic.go:356] "Generic (PLEG): container finished" podID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerID="93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463" exitCode=0
Sep 09 21:13:33.509541 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509134    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"
Sep 09 21:13:33.509541 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerDied","Data":"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"}
Sep 09 21:13:33.509541 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh" event={"ID":"db874311-ea63-403c-b05f-c0668bfbbf1f","Type":"ContainerDied","Data":"b2118d0dfa4f8b75b461f90fbefe25443134d8a9df8bacf75279857a70574568"}
Sep 09 21:13:33.509541 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509250    2578 scope.go:117] "RemoveContainer" containerID="444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"
Sep 09 21:13:33.510304 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:13:33.510304 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.509964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:13:33.519521 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.519505    2578 scope.go:117] "RemoveContainer" containerID="93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"
Sep 09 21:13:33.526941 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.526925    2578 scope.go:117] "RemoveContainer" containerID="444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"
Sep 09 21:13:33.527217 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:33.527199    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c\": container with ID starting with 444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c not found: ID does not exist" containerID="444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"
Sep 09 21:13:33.527267 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.527225    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c"} err="failed to get container status \"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c\": rpc error: code = NotFound desc = could not find container \"444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c\": container with ID starting with 444065780b4da119b29b3156aff2625a70838b9cedac6c8ad28d9d68b2c0bc7c not found: ID does not exist"
Sep 09 21:13:33.527267 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.527241    2578 scope.go:117] "RemoveContainer" containerID="93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"
Sep 09 21:13:33.527444 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:33.527429    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463\": container with ID starting with 93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463 not found: ID does not exist" containerID="93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"
Sep 09 21:13:33.527482 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.527450    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463"} err="failed to get container status \"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463\": rpc error: code = NotFound desc = could not find container \"93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463\": container with ID starting with 93d7dc9de2070115947ebe8e22fd2337d265b534aa513a65e5f9553c81e6a463 not found: ID does not exist"
Sep 09 21:13:33.535559 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.535538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:13:33.539194 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:33.539177    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac5-predictor-59646b8c6c-2dhxh"]
Sep 09 21:13:35.284209 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:35.284178    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" path="/var/lib/kubelet/pods/db874311-ea63-403c-b05f-c0668bfbbf1f/volumes"
Sep 09 21:13:37.153844 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:37.153802    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 09 21:13:37.159154 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:37.159120    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 09 21:13:37.353979 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:37.353936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 09 21:13:37.354373 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:37.354338    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 09 21:13:38.054904 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.054881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:13:38.098930 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.098898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8llv\" (UniqueName: \"kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv\") pod \"b5fab6ce-1fa3-43a9-8345-04945b21d047\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") "
Sep 09 21:13:38.099098 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.098992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config\") pod \"b5fab6ce-1fa3-43a9-8345-04945b21d047\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") "
Sep 09 21:13:38.099098 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.099047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") pod \"b5fab6ce-1fa3-43a9-8345-04945b21d047\" (UID: \"b5fab6ce-1fa3-43a9-8345-04945b21d047\") "
Sep 09 21:13:38.099371 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.099342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5fab6ce-1fa3-43a9-8345-04945b21d047" (UID: "b5fab6ce-1fa3-43a9-8345-04945b21d047"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:38.101136 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.101104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv" (OuterVolumeSpecName: "kube-api-access-m8llv") pod "b5fab6ce-1fa3-43a9-8345-04945b21d047" (UID: "b5fab6ce-1fa3-43a9-8345-04945b21d047"). InnerVolumeSpecName "kube-api-access-m8llv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:38.101136 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.101129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5fab6ce-1fa3-43a9-8345-04945b21d047" (UID: "b5fab6ce-1fa3-43a9-8345-04945b21d047"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:38.199884 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.199808    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fab6ce-1fa3-43a9-8345-04945b21d047-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:38.199884 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.199830    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8llv\" (UniqueName: \"kubernetes.io/projected/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-api-access-m8llv\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:38.199884 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.199841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fab6ce-1fa3-43a9-8345-04945b21d047-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:13:38.514258 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.514165    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:13:38.515195 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.514823    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:13:38.515346 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.515248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:13:38.515727 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.515699    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:13:38.537845 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.537816    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerID="f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea" exitCode=0
Sep 09 21:13:38.537845 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.537830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerDied","Data":"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"}
Sep 09 21:13:38.538077 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.537875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv" event={"ID":"b5fab6ce-1fa3-43a9-8345-04945b21d047","Type":"ContainerDied","Data":"f0e40d813717bda3ef653a80bf82b3671f2f4cc79a04e702d240f4f6607de065"}
Sep 09 21:13:38.538077 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.537900    2578 scope.go:117] "RemoveContainer" containerID="242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"
Sep 09 21:13:38.538077 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.537931    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"
Sep 09 21:13:38.546508 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.546490    2578 scope.go:117] "RemoveContainer" containerID="f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"
Sep 09 21:13:38.553693 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.553678    2578 scope.go:117] "RemoveContainer" containerID="242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"
Sep 09 21:13:38.553926 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:38.553911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744\": container with ID starting with 242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744 not found: ID does not exist" containerID="242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"
Sep 09 21:13:38.553981 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.553935    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744"} err="failed to get container status \"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744\": rpc error: code = NotFound desc = could not find container \"242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744\": container with ID starting with 242fb591b90ce801defcf1dbf11606c7260f4f55dd31d64130f449c346424744 not found: ID does not exist"
Sep 09 21:13:38.553981 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.553957    2578 scope.go:117] "RemoveContainer" containerID="f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"
Sep 09 21:13:38.554170 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:13:38.554152    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea\": container with ID starting with f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea not found: ID does not exist" containerID="f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"
Sep 09 21:13:38.554235 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.554180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea"} err="failed to get container status \"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea\": rpc error: code = NotFound desc = could not find container \"f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea\": container with ID starting with f80261935fec68694a39fd836139595163debc41c429b86bac4d69b5365d70ea not found: ID does not exist"
Sep 09 21:13:38.571958 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.571930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:13:38.575982 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:38.575957    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac5-predictor-7775796c74-h95dv"]
Sep 09 21:13:39.284223 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:39.284183    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" path="/var/lib/kubelet/pods/b5fab6ce-1fa3-43a9-8345-04945b21d047/volumes"
Sep 09 21:13:47.354794 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:47.354741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:13:47.355165 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:47.354855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:13:48.514764 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:48.514705    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:13:48.516186 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:48.516162    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:13:58.514869 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:58.514831    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:13:58.516351 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:13:58.516332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:14:08.514808 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:08.514774    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:14:08.515924 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:08.515901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:14:09.338095 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.338063    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:14:09.338319 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.338294    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container" containerID="cri-o://710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c" gracePeriod=30
Sep 09 21:14:09.338379 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.338332    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kube-rbac-proxy" containerID="cri-o://819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47" gracePeriod=30
Sep 09 21:14:09.497224 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.497190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:14:09.497412 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.497391    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container" containerID="cri-o://7cdf2aa47f213f1c8b1643b600427d8048eb2d96fb60b16b4992ca33e631ae5e" gracePeriod=30
Sep 09 21:14:09.497537 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.497443    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kube-rbac-proxy" containerID="cri-o://afaafaa7677d61524c9b4782282c0b034c06bf838c8fa28b2448b93f9fbcd5fa" gracePeriod=30
Sep 09 21:14:09.520134 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520107    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:14:09.520440 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520428    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container"
Sep 09 21:14:09.520492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container"
Sep 09 21:14:09.520492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520470    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kube-rbac-proxy"
Sep 09 21:14:09.520492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520476    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kube-rbac-proxy"
Sep 09 21:14:09.520492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520482    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy"
Sep 09 21:14:09.520492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520487    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520496    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520501    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520565    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kserve-container"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kserve-container"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520579    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fab6ce-1fa3-43a9-8345-04945b21d047" containerName="kube-rbac-proxy"
Sep 09 21:14:09.520640 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.520585    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db874311-ea63-403c-b05f-c0668bfbbf1f" containerName="kube-rbac-proxy"
Sep 09 21:14:09.523707 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.523692    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.529949 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.529848    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dc3b-predictor-serving-cert\""
Sep 09 21:14:09.533628 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.533613    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dc3b-kube-rbac-proxy-sar-config\""
Sep 09 21:14:09.550580 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.550556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:14:09.642910 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.642873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.642910 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.642912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58j9r\" (UniqueName: \"kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.643096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.642939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.672973 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.672944    2578 generic.go:356] "Generic (PLEG): container finished" podID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerID="819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47" exitCode=2
Sep 09 21:14:09.673144 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.673022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerDied","Data":"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"}
Sep 09 21:14:09.674940 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.674916    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerID="afaafaa7677d61524c9b4782282c0b034c06bf838c8fa28b2448b93f9fbcd5fa" exitCode=2
Sep 09 21:14:09.675034 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.674990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerDied","Data":"afaafaa7677d61524c9b4782282c0b034c06bf838c8fa28b2448b93f9fbcd5fa"}
Sep 09 21:14:09.708137 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.708100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:14:09.711422 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.711403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.716095 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.716075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dc3b-predictor-serving-cert\""
Sep 09 21:14:09.716424 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.716405    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dc3b-kube-rbac-proxy-sar-config\""
Sep 09 21:14:09.734588 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.734560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:14:09.744285 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.744261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.744409 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.744296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58j9r\" (UniqueName: \"kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.744409 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.744324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.744984 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.744965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.746787 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.746766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.762041 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.762017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58j9r\" (UniqueName: \"kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r\") pod \"success-200-isvc-0dc3b-predictor-56f6689897-dw9g7\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") " pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.832939 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.832911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:09.845066 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.845033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.845165 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.845100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jm5\" (UniqueName: \"kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.845165 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.845153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.946232 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.946202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jm5\" (UniqueName: \"kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.946388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.946245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.946388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.946305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.946922 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.946902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.948823 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.948807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:09.955991 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:09.955975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jm5\" (UniqueName: \"kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5\") pod \"error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") " pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:10.021671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.021636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:10.110830 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.110802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:14:10.112814 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:14:10.112787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550b3445_4d15_44ad_bb94_eef6889630ee.slice/crio-d925d375dd93028409c9365324552f261b1f9520e5429e37b5b540f680c2d44d WatchSource:0}: Error finding container d925d375dd93028409c9365324552f261b1f9520e5429e37b5b540f680c2d44d: Status 404 returned error can't find the container with id d925d375dd93028409c9365324552f261b1f9520e5429e37b5b540f680c2d44d
Sep 09 21:14:10.294619 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.294595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:14:10.296886 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:14:10.296862    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadf395b_07b2_4e62_b0e1_083f927492db.slice/crio-807b1ae8472e5ac5f099bd116e5ecfc99999f71555ce1dee2fd1fe8d6beb23ed WatchSource:0}: Error finding container 807b1ae8472e5ac5f099bd116e5ecfc99999f71555ce1dee2fd1fe8d6beb23ed: Status 404 returned error can't find the container with id 807b1ae8472e5ac5f099bd116e5ecfc99999f71555ce1dee2fd1fe8d6beb23ed
Sep 09 21:14:10.683804 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.683711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerStarted","Data":"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"}
Sep 09 21:14:10.684213 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.683805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerStarted","Data":"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"}
Sep 09 21:14:10.684213 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.683837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerStarted","Data":"807b1ae8472e5ac5f099bd116e5ecfc99999f71555ce1dee2fd1fe8d6beb23ed"}
Sep 09 21:14:10.684926 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.684790    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:10.684926 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.684880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:10.686688 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.686462    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:10.687679 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.687650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerStarted","Data":"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"}
Sep 09 21:14:10.687806 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.687708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerStarted","Data":"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"}
Sep 09 21:14:10.687806 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.687735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerStarted","Data":"d925d375dd93028409c9365324552f261b1f9520e5429e37b5b540f680c2d44d"}
Sep 09 21:14:10.688060 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.688040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:10.688144 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.688088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:10.689248 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.689212    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:10.706369 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:10.706330    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podStartSLOduration=1.706316377 podStartE2EDuration="1.706316377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:10.705010727 +0000 UTC m=+1297.870338006" watchObservedRunningTime="2026-09-09 21:14:10.706316377 +0000 UTC m=+1297.871643699"
Sep 09 21:14:11.691953 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:11.691917    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:11.692345 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:11.692020    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:12.349683 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.349647    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 09 21:14:12.349869 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.349647    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 09 21:14:12.698479 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.698402    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerID="7cdf2aa47f213f1c8b1643b600427d8048eb2d96fb60b16b4992ca33e631ae5e" exitCode=0
Sep 09 21:14:12.698846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.698476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerDied","Data":"7cdf2aa47f213f1c8b1643b600427d8048eb2d96fb60b16b4992ca33e631ae5e"}
Sep 09 21:14:12.698899 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.698853    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:12.995000 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:12.994973    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:14:13.017694 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.017648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podStartSLOduration=4.017632659 podStartE2EDuration="4.017632659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:10.731530985 +0000 UTC m=+1297.896858253" watchObservedRunningTime="2026-09-09 21:14:13.017632659 +0000 UTC m=+1300.182959926"
Sep 09 21:14:13.038395 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.038378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:14:13.071809 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.071786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls\") pod \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") "
Sep 09 21:14:13.071905 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.071842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptqgj\" (UniqueName: \"kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj\") pod \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") "
Sep 09 21:14:13.071905 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.071873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config\") pod \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\" (UID: \"12c4adf6-9ca1-494d-a53a-588ece6f8d98\") "
Sep 09 21:14:13.072234 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.072214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12c4adf6-9ca1-494d-a53a-588ece6f8d98" (UID: "12c4adf6-9ca1-494d-a53a-588ece6f8d98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:13.073924 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.073894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12c4adf6-9ca1-494d-a53a-588ece6f8d98" (UID: "12c4adf6-9ca1-494d-a53a-588ece6f8d98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:13.074051 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.074036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj" (OuterVolumeSpecName: "kube-api-access-ptqgj") pod "12c4adf6-9ca1-494d-a53a-588ece6f8d98" (UID: "12c4adf6-9ca1-494d-a53a-588ece6f8d98"). InnerVolumeSpecName "kube-api-access-ptqgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:13.173217 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") pod \"8fd0fce0-66cc-4939-9d3c-27ac27674902\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") "
Sep 09 21:14:13.173344 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx\") pod \"8fd0fce0-66cc-4939-9d3c-27ac27674902\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") "
Sep 09 21:14:13.173344 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config\") pod \"8fd0fce0-66cc-4939-9d3c-27ac27674902\" (UID: \"8fd0fce0-66cc-4939-9d3c-27ac27674902\") "
Sep 09 21:14:13.173508 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.173558 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173508    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c4adf6-9ca1-494d-a53a-588ece6f8d98-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.173558 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173523    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqgj\" (UniqueName: \"kubernetes.io/projected/12c4adf6-9ca1-494d-a53a-588ece6f8d98-kube-api-access-ptqgj\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.173666 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.173645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fd0fce0-66cc-4939-9d3c-27ac27674902" (UID: "8fd0fce0-66cc-4939-9d3c-27ac27674902"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:13.175144 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.175121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fd0fce0-66cc-4939-9d3c-27ac27674902" (UID: "8fd0fce0-66cc-4939-9d3c-27ac27674902"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:13.175228 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.175213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx" (OuterVolumeSpecName: "kube-api-access-v8zvx") pod "8fd0fce0-66cc-4939-9d3c-27ac27674902" (UID: "8fd0fce0-66cc-4939-9d3c-27ac27674902"). InnerVolumeSpecName "kube-api-access-v8zvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:13.274768 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.274679    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-api-access-v8zvx\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.274768 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.274705    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fce0-66cc-4939-9d3c-27ac27674902-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.274768 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.274718    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fce0-66cc-4939-9d3c-27ac27674902-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:14:13.708562 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.708525    2578 generic.go:356] "Generic (PLEG): container finished" podID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerID="710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c" exitCode=0
Sep 09 21:14:13.709063 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.708597    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"
Sep 09 21:14:13.709063 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.708593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerDied","Data":"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"}
Sep 09 21:14:13.709063 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.708800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx" event={"ID":"12c4adf6-9ca1-494d-a53a-588ece6f8d98","Type":"ContainerDied","Data":"cd54760ad8261643582028623588b139498e8cbf5e0572f886d0c57debce5a61"}
Sep 09 21:14:13.709063 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.708832    2578 scope.go:117] "RemoveContainer" containerID="819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"
Sep 09 21:14:13.710542 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.710464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm" event={"ID":"8fd0fce0-66cc-4939-9d3c-27ac27674902","Type":"ContainerDied","Data":"a4e52614c7b2d5d2e102ad6ac14988b6b2644cea900c50875ae5c87ff7a77a2a"}
Sep 09 21:14:13.710651 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.710552    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"
Sep 09 21:14:13.717418 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.717402    2578 scope.go:117] "RemoveContainer" containerID="710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"
Sep 09 21:14:13.724894 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.724878    2578 scope.go:117] "RemoveContainer" containerID="819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"
Sep 09 21:14:13.725134 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:14:13.725114    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47\": container with ID starting with 819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47 not found: ID does not exist" containerID="819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"
Sep 09 21:14:13.725187 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.725140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47"} err="failed to get container status \"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47\": rpc error: code = NotFound desc = could not find container \"819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47\": container with ID starting with 819158d871d1796cad6057e9df11019eca64d55cc31962c83e650a3afa2b9e47 not found: ID does not exist"
Sep 09 21:14:13.725187 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.725156    2578 scope.go:117] "RemoveContainer" containerID="710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"
Sep 09 21:14:13.725399 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:14:13.725384    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c\": container with ID starting with 710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c not found: ID does not exist" containerID="710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"
Sep 09 21:14:13.725436 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.725406    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c"} err="failed to get container status \"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c\": rpc error: code = NotFound desc = could not find container \"710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c\": container with ID starting with 710cb7426845db38fa948e0b87255a552f49f2c27ebf8f66f21085189135b98c not found: ID does not exist"
Sep 09 21:14:13.725436 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.725419    2578 scope.go:117] "RemoveContainer" containerID="afaafaa7677d61524c9b4782282c0b034c06bf838c8fa28b2448b93f9fbcd5fa"
Sep 09 21:14:13.730646 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.730629    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:14:13.732965 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.732949    2578 scope.go:117] "RemoveContainer" containerID="7cdf2aa47f213f1c8b1643b600427d8048eb2d96fb60b16b4992ca33e631ae5e"
Sep 09 21:14:13.735611 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.735594    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07fe8-predictor-556db9dbb5-sl5dx"]
Sep 09 21:14:13.751501 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.751483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:14:13.757359 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:13.757341    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07fe8-predictor-d74cf8d9b-tnlbm"]
Sep 09 21:14:15.284934 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:15.284897    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" path="/var/lib/kubelet/pods/12c4adf6-9ca1-494d-a53a-588ece6f8d98/volumes"
Sep 09 21:14:15.285535 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:15.285513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" path="/var/lib/kubelet/pods/8fd0fce0-66cc-4939-9d3c-27ac27674902/volumes"
Sep 09 21:14:16.696366 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:16.696341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:16.696809 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:16.696784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:17.703009 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:17.702976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:14:17.703457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:17.703433    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:18.514946 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:18.514915    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 09 21:14:18.516128 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:18.516104    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 09 21:14:26.696947 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:26.696906    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:27.703796 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:27.703739    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:28.515518 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:28.515491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:14:28.516326 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:28.516310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:14:36.697142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:36.697105    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:37.703800 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:37.703765    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:46.697645 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:46.697601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:14:47.703976 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:47.703938    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:14:56.697950 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:56.697881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:14:57.703886 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:14:57.703858    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:22:54.942695 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.942662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:22:54.943156 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.942884    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container" containerID="cri-o://41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa" gracePeriod=30
Sep 09 21:22:54.943156 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.942943    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kube-rbac-proxy" containerID="cri-o://abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db" gracePeriod=30
Sep 09 21:22:54.984424 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984392    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:22:54.984788 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984770    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984791    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984809    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984819    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kube-rbac-proxy"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984847    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kube-rbac-proxy"
Sep 09 21:22:54.984877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984873    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kube-rbac-proxy"
Sep 09 21:22:54.985166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984881    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kube-rbac-proxy"
Sep 09 21:22:54.985166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984968    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kube-rbac-proxy"
Sep 09 21:22:54.985166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984979    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd0fce0-66cc-4939-9d3c-27ac27674902" containerName="kserve-container"
Sep 09 21:22:54.985166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.984991    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kube-rbac-proxy"
Sep 09 21:22:54.985166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.985002    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c4adf6-9ca1-494d-a53a-588ece6f8d98" containerName="kserve-container"
Sep 09 21:22:54.988220 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.988188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:54.991202 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.991179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78161-predictor-serving-cert\""
Sep 09 21:22:54.991318 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:54.991247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78161-kube-rbac-proxy-sar-config\""
Sep 09 21:22:55.004987 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.004957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.005096 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.005060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6c7\" (UniqueName: \"kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.005161 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.005092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.005474 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.005452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:22:55.050080 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.050048    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:22:55.050503 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.050477    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container" containerID="cri-o://bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e" gracePeriod=30
Sep 09 21:22:55.050785 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.050523    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kube-rbac-proxy" containerID="cri-o://48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561" gracePeriod=30
Sep 09 21:22:55.105811 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.105783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.105936 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.105838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6c7\" (UniqueName: \"kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.105936 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.105865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.106042 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.106018    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-78161-predictor-serving-cert: secret "success-200-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.106094 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.106069    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls podName:18e55106-7ad0-415c-8298-1fc1b753b0ec nodeName:}" failed. No retries permitted until 2026-09-09 21:22:55.606054192 +0000 UTC m=+1822.771381438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls") pod "success-200-isvc-78161-predictor-6cfdb6f587-twf6p" (UID: "18e55106-7ad0-415c-8298-1fc1b753b0ec") : secret "success-200-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.106422 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.106401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.123454 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.123424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6c7\" (UniqueName: \"kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.131616 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.131595    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:22:55.135430 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.135415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.138381 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.138329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78161-predictor-serving-cert\""
Sep 09 21:22:55.138767 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.138732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78161-kube-rbac-proxy-sar-config\""
Sep 09 21:22:55.148440 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.148418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:22:55.207094 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.207026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.207094 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.207060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jcq\" (UniqueName: \"kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.207244 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.207143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.308141 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.308114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.308274 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.308147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jcq\" (UniqueName: \"kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.308326 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.308271    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-78161-predictor-serving-cert: secret "error-404-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.308326 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.308291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.308395 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.308336    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls podName:47ac90c1-e488-44bd-ac79-a387359b50d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:22:55.808315787 +0000 UTC m=+1822.973643038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls") pod "error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" (UID: "47ac90c1-e488-44bd-ac79-a387359b50d4") : secret "error-404-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.309017 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.308997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.320164 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.320135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jcq\" (UniqueName: \"kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.611145 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.611114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:55.611307 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.611222    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-78161-predictor-serving-cert: secret "success-200-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.611307 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.611280    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls podName:18e55106-7ad0-415c-8298-1fc1b753b0ec nodeName:}" failed. No retries permitted until 2026-09-09 21:22:56.611266823 +0000 UTC m=+1823.776594069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls") pod "success-200-isvc-78161-predictor-6cfdb6f587-twf6p" (UID: "18e55106-7ad0-415c-8298-1fc1b753b0ec") : secret "success-200-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.812841 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.812801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:55.813019 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.812940    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-78161-predictor-serving-cert: secret "error-404-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.813089 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:55.813020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls podName:47ac90c1-e488-44bd-ac79-a387359b50d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:22:56.81300222 +0000 UTC m=+1823.978329470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls") pod "error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" (UID: "47ac90c1-e488-44bd-ac79-a387359b50d4") : secret "error-404-isvc-78161-predictor-serving-cert" not found
Sep 09 21:22:55.891531 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.891441    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerID="abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db" exitCode=2
Sep 09 21:22:55.891531 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.891511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerDied","Data":"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"}
Sep 09 21:22:55.893412 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.893391    2578 generic.go:356] "Generic (PLEG): container finished" podID="5694902d-0f73-4a27-9a32-855abbf57462" containerID="48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561" exitCode=2
Sep 09 21:22:55.893543 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:55.893446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerDied","Data":"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"}
Sep 09 21:22:56.618513 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.618475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:56.621041 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.621022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"success-200-isvc-78161-predictor-6cfdb6f587-twf6p\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") " pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:56.801461 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.801425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:56.820679 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.820654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:56.823151 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.823130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"error-404-isvc-78161-predictor-74f5dcc9c4-lts9z\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") " pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:56.945735 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:56.945657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:57.142784 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.140665    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:22:57.248142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.248121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:22:57.250961 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:22:57.250934    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ac90c1_e488_44bd_ac79_a387359b50d4.slice/crio-0239f0036e92980f2c644be1296f9201ad21af34332fe620d6c434be3df3c6f1 WatchSource:0}: Error finding container 0239f0036e92980f2c644be1296f9201ad21af34332fe620d6c434be3df3c6f1: Status 404 returned error can't find the container with id 0239f0036e92980f2c644be1296f9201ad21af34332fe620d6c434be3df3c6f1
Sep 09 21:22:57.906951 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.906920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerStarted","Data":"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"}
Sep 09 21:22:57.907284 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.906986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerStarted","Data":"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"}
Sep 09 21:22:57.907284 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.907014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerStarted","Data":"0239f0036e92980f2c644be1296f9201ad21af34332fe620d6c434be3df3c6f1"}
Sep 09 21:22:57.907284 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.907151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:57.908759 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.908719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerStarted","Data":"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"}
Sep 09 21:22:57.908866 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.908799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:57.908866 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.908833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerStarted","Data":"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"}
Sep 09 21:22:57.908866 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.908860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerStarted","Data":"99554b94af758ef5aa59379d0010da682ad355144675ea1e6d9c64be8c63095a"}
Sep 09 21:22:57.909017 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.908894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:22:57.910268 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.910246    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:22:57.928664 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.928623    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podStartSLOduration=2.928610828 podStartE2EDuration="2.928610828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:57.926906135 +0000 UTC m=+1825.092233403" watchObservedRunningTime="2026-09-09 21:22:57.928610828 +0000 UTC m=+1825.093938095"
Sep 09 21:22:57.947709 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:57.947673    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podStartSLOduration=3.94766026 podStartE2EDuration="3.94766026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:57.946057977 +0000 UTC m=+1825.111385244" watchObservedRunningTime="2026-09-09 21:22:57.94766026 +0000 UTC m=+1825.112987527"
Sep 09 21:22:58.110085 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.110062    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:22:58.112793 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.112776    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:22:58.233244 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g4k9\" (UniqueName: \"kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9\") pod \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") "
Sep 09 21:22:58.233244 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp\") pod \"5694902d-0f73-4a27-9a32-855abbf57462\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") "
Sep 09 21:22:58.233430 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233264    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config\") pod \"5694902d-0f73-4a27-9a32-855abbf57462\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") "
Sep 09 21:22:58.233430 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") pod \"5694902d-0f73-4a27-9a32-855abbf57462\" (UID: \"5694902d-0f73-4a27-9a32-855abbf57462\") "
Sep 09 21:22:58.233430 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") pod \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") "
Sep 09 21:22:58.233592 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config\") pod \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\" (UID: \"ad98443a-63a8-45d9-8e57-e23ba7cc1779\") "
Sep 09 21:22:58.233716 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5694902d-0f73-4a27-9a32-855abbf57462" (UID: "5694902d-0f73-4a27-9a32-855abbf57462"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:58.233852 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.233826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad98443a-63a8-45d9-8e57-e23ba7cc1779" (UID: "ad98443a-63a8-45d9-8e57-e23ba7cc1779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:58.235495 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.235467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad98443a-63a8-45d9-8e57-e23ba7cc1779" (UID: "ad98443a-63a8-45d9-8e57-e23ba7cc1779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:58.235591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.235541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5694902d-0f73-4a27-9a32-855abbf57462" (UID: "5694902d-0f73-4a27-9a32-855abbf57462"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:58.235591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.235563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9" (OuterVolumeSpecName: "kube-api-access-4g4k9") pod "ad98443a-63a8-45d9-8e57-e23ba7cc1779" (UID: "ad98443a-63a8-45d9-8e57-e23ba7cc1779"). InnerVolumeSpecName "kube-api-access-4g4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:58.235591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.235575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp" (OuterVolumeSpecName: "kube-api-access-bfxqp") pod "5694902d-0f73-4a27-9a32-855abbf57462" (UID: "5694902d-0f73-4a27-9a32-855abbf57462"). InnerVolumeSpecName "kube-api-access-bfxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:58.334565 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334532    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/5694902d-0f73-4a27-9a32-855abbf57462-kube-api-access-bfxqp\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.334671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334628    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5694902d-0f73-4a27-9a32-855abbf57462-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.334671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334639    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5694902d-0f73-4a27-9a32-855abbf57462-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.334671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334648    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad98443a-63a8-45d9-8e57-e23ba7cc1779-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.334671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334655    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.334671 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.334664    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4k9\" (UniqueName: \"kubernetes.io/projected/ad98443a-63a8-45d9-8e57-e23ba7cc1779-kube-api-access-4g4k9\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:22:58.914871 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.914794    2578 generic.go:356] "Generic (PLEG): container finished" podID="5694902d-0f73-4a27-9a32-855abbf57462" containerID="bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e" exitCode=0
Sep 09 21:22:58.914871 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.914849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"
Sep 09 21:22:58.915325 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.914875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerDied","Data":"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"}
Sep 09 21:22:58.915325 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.914929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8" event={"ID":"5694902d-0f73-4a27-9a32-855abbf57462","Type":"ContainerDied","Data":"9c7a1095de6c4fdbfc6584325cfb2d910dca0c0c9e9043786b5e361002fe7a66"}
Sep 09 21:22:58.915325 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.914956    2578 scope.go:117] "RemoveContainer" containerID="48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"
Sep 09 21:22:58.916883 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.916862    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerID="41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa" exitCode=0
Sep 09 21:22:58.916963 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.916914    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"
Sep 09 21:22:58.916963 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.916939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerDied","Data":"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"}
Sep 09 21:22:58.917051 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.917003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6" event={"ID":"ad98443a-63a8-45d9-8e57-e23ba7cc1779","Type":"ContainerDied","Data":"6aeae0874b8230d20ffddff69e24bbe4b9a5b02ca3937b0e5190f71d08584774"}
Sep 09 21:22:58.917382 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.917357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:22:58.917690 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.917664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:22:58.919298 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.919265    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:22:58.925197 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.925176    2578 scope.go:117] "RemoveContainer" containerID="bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"
Sep 09 21:22:58.934881 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.934865    2578 scope.go:117] "RemoveContainer" containerID="48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"
Sep 09 21:22:58.935139 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:58.935118    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561\": container with ID starting with 48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561 not found: ID does not exist" containerID="48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"
Sep 09 21:22:58.935206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.935150    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561"} err="failed to get container status \"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561\": rpc error: code = NotFound desc = could not find container \"48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561\": container with ID starting with 48f944a924334b99dbd48f46b6dbbf7e16309f75aa5f2c8cb71318d6030ff561 not found: ID does not exist"
Sep 09 21:22:58.935206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.935175    2578 scope.go:117] "RemoveContainer" containerID="bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"
Sep 09 21:22:58.935454 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:58.935438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e\": container with ID starting with bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e not found: ID does not exist" containerID="bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"
Sep 09 21:22:58.935494 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.935460    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e"} err="failed to get container status \"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e\": rpc error: code = NotFound desc = could not find container \"bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e\": container with ID starting with bcac812fe986af7b73c76ed7764f75bf164a48fa5d6593205377cbcf1844371e not found: ID does not exist"
Sep 09 21:22:58.935494 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.935474    2578 scope.go:117] "RemoveContainer" containerID="abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"
Sep 09 21:22:58.942353 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.942336    2578 scope.go:117] "RemoveContainer" containerID="41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"
Sep 09 21:22:58.945943 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.945924    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:22:58.949657 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.949642    2578 scope.go:117] "RemoveContainer" containerID="abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"
Sep 09 21:22:58.949921 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:58.949902    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db\": container with ID starting with abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db not found: ID does not exist" containerID="abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"
Sep 09 21:22:58.949989 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.949930    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db"} err="failed to get container status \"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db\": rpc error: code = NotFound desc = could not find container \"abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db\": container with ID starting with abcb9670d6b16fbe6a671f1d3c0308e4ec84e8779abb55ad03d154e1523067db not found: ID does not exist"
Sep 09 21:22:58.949989 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.949950    2578 scope.go:117] "RemoveContainer" containerID="41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"
Sep 09 21:22:58.950206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.950087    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e5cb-predictor-bcc98846-sthd8"]
Sep 09 21:22:58.950206 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:22:58.950170    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa\": container with ID starting with 41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa not found: ID does not exist" containerID="41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"
Sep 09 21:22:58.950206 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.950195    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa"} err="failed to get container status \"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa\": rpc error: code = NotFound desc = could not find container \"41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa\": container with ID starting with 41dcf87cabceaf2ac64d6e554bda47554f7a773118a120778a49a2f035754caa not found: ID does not exist"
Sep 09 21:22:58.962610 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.962588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:22:58.970974 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:58.970958    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e5cb-predictor-5766cb6459-k46x6"]
Sep 09 21:22:59.283698 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:59.283619    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5694902d-0f73-4a27-9a32-855abbf57462" path="/var/lib/kubelet/pods/5694902d-0f73-4a27-9a32-855abbf57462/volumes"
Sep 09 21:22:59.284264 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:59.284244    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" path="/var/lib/kubelet/pods/ad98443a-63a8-45d9-8e57-e23ba7cc1779/volumes"
Sep 09 21:22:59.923430 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:22:59.923394    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:23:03.922163 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:03.922135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:23:03.922606 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:03.922490    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:23:04.928088 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:04.928059    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:23:04.928502 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:04.928477    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:23:13.923176 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:13.923131    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:23:14.929419 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:14.929378    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:23:23.922553 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:23.922504    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:23:24.242983 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.242904    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:23:24.243153 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.243110    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" containerID="cri-o://f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919" gracePeriod=30
Sep 09 21:23:24.243243 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.243128    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kube-rbac-proxy" containerID="cri-o://6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9" gracePeriod=30
Sep 09 21:23:24.312484 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.312444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:23:24.312715 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.312690    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" containerID="cri-o://9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105" gracePeriod=30
Sep 09 21:23:24.312806 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.312710    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kube-rbac-proxy" containerID="cri-o://9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd" gracePeriod=30
Sep 09 21:23:24.423405 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:23:24.423690 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423676    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container"
Sep 09 21:23:24.423690 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423691    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423705    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423714    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kube-rbac-proxy"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423727    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kube-rbac-proxy"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423757    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kube-rbac-proxy"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423766    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kube-rbac-proxy"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423819    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kserve-container"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423827    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kserve-container"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423837    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5694902d-0f73-4a27-9a32-855abbf57462" containerName="kube-rbac-proxy"
Sep 09 21:23:24.423846 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.423847    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad98443a-63a8-45d9-8e57-e23ba7cc1779" containerName="kube-rbac-proxy"
Sep 09 21:23:24.430192 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.430165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.433734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.433499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-72d00-predictor-serving-cert\""
Sep 09 21:23:24.433734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.433674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-72d00-kube-rbac-proxy-sar-config\""
Sep 09 21:23:24.440190 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.440169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:23:24.527038 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.526966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzgk\" (UniqueName: \"kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.527038 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.527012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.527240 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.527112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.560607 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.560580    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:23:24.563931 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.563915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.566295 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.566267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-72d00-predictor-serving-cert\""
Sep 09 21:23:24.566404 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.566379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-72d00-kube-rbac-proxy-sar-config\""
Sep 09 21:23:24.586770 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.577908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:23:24.628356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.628497 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.628497 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdjj\" (UniqueName: \"kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.628606 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.628658 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:24.628615    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-72d00-predictor-serving-cert: secret "success-200-isvc-72d00-predictor-serving-cert" not found
Sep 09 21:23:24.628711 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.628711 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:24.628674    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls podName:0c0a5fbe-bf23-44cb-a836-530405a30899 nodeName:}" failed. No retries permitted until 2026-09-09 21:23:25.128655315 +0000 UTC m=+1852.293982580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls") pod "success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" (UID: "0c0a5fbe-bf23-44cb-a836-530405a30899") : secret "success-200-isvc-72d00-predictor-serving-cert" not found
Sep 09 21:23:24.628711 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzgk\" (UniqueName: \"kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.628997 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.628980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.640170 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.640151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzgk\" (UniqueName: \"kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:24.729306 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.729262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdjj\" (UniqueName: \"kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.729586 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.729344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.729586 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.729394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.729586 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:24.729447    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-72d00-predictor-serving-cert: secret "error-404-isvc-72d00-predictor-serving-cert" not found
Sep 09 21:23:24.729586 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:24.729516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls podName:3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43 nodeName:}" failed. No retries permitted until 2026-09-09 21:23:25.229498328 +0000 UTC m=+1852.394825577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls") pod "error-404-isvc-72d00-predictor-54876486fd-sldbf" (UID: "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43") : secret "error-404-isvc-72d00-predictor-serving-cert" not found
Sep 09 21:23:24.730183 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.730160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.739356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.739334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdjj\" (UniqueName: \"kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:24.929156 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:24.929123    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:23:25.030054 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.030026    2578 generic.go:356] "Generic (PLEG): container finished" podID="fadf395b-07b2-4e62-b0e1-083f927492db" containerID="9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd" exitCode=2
Sep 09 21:23:25.030229 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.030085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerDied","Data":"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"}
Sep 09 21:23:25.031812 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.031789    2578 generic.go:356] "Generic (PLEG): container finished" podID="550b3445-4d15-44ad-bb94-eef6889630ee" containerID="6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9" exitCode=2
Sep 09 21:23:25.031908 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.031851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerDied","Data":"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"}
Sep 09 21:23:25.132948 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.132922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:25.135410 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.135392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") pod \"success-200-isvc-72d00-predictor-59d5cd5987-7hrxk\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") " pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:25.233348 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.233274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:25.235639 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.235616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") pod \"error-404-isvc-72d00-predictor-54876486fd-sldbf\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") " pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:25.342762 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.342718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:25.476142 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.476112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:25.625942 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.625910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:23:25.629947 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:23:25.629923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0a5fbe_bf23_44cb_a836_530405a30899.slice/crio-d8dd682fd3d01445bca10dd50626fbf10c59dabfda5a373f0df41e6abf8e8d7b WatchSource:0}: Error finding container d8dd682fd3d01445bca10dd50626fbf10c59dabfda5a373f0df41e6abf8e8d7b: Status 404 returned error can't find the container with id d8dd682fd3d01445bca10dd50626fbf10c59dabfda5a373f0df41e6abf8e8d7b
Sep 09 21:23:25.753592 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:25.753565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:23:25.755378 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:23:25.755354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee610c2_dc5e_43b8_bf7b_1a67f6e63c43.slice/crio-a10f18e8f78bcda2c666b91f1190c1b05d161d9547279907d190d10fdcafd19b WatchSource:0}: Error finding container a10f18e8f78bcda2c666b91f1190c1b05d161d9547279907d190d10fdcafd19b: Status 404 returned error can't find the container with id a10f18e8f78bcda2c666b91f1190c1b05d161d9547279907d190d10fdcafd19b
Sep 09 21:23:26.037964 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.037874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerStarted","Data":"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"}
Sep 09 21:23:26.037964 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.037944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:26.038410 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.038092    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:26.038410 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.038147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerStarted","Data":"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"}
Sep 09 21:23:26.038410 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.038181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerStarted","Data":"a10f18e8f78bcda2c666b91f1190c1b05d161d9547279907d190d10fdcafd19b"}
Sep 09 21:23:26.039653 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.039626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:23:26.039880 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.039857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerStarted","Data":"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"}
Sep 09 21:23:26.039946 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.039908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerStarted","Data":"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"}
Sep 09 21:23:26.039946 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.039941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerStarted","Data":"d8dd682fd3d01445bca10dd50626fbf10c59dabfda5a373f0df41e6abf8e8d7b"}
Sep 09 21:23:26.040037 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.039973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:26.061531 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.061490    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podStartSLOduration=2.061477676 podStartE2EDuration="2.061477676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:26.059882818 +0000 UTC m=+1853.225210112" watchObservedRunningTime="2026-09-09 21:23:26.061477676 +0000 UTC m=+1853.226805019"
Sep 09 21:23:26.092310 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.092265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podStartSLOduration=2.092250609 podStartE2EDuration="2.092250609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:26.089949026 +0000 UTC m=+1853.255276294" watchObservedRunningTime="2026-09-09 21:23:26.092250609 +0000 UTC m=+1853.257577878"
Sep 09 21:23:26.692474 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.692428    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 09 21:23:26.697224 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:26.697199    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 09 21:23:27.044249 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.044160    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:23:27.044249 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.044191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:27.045492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.045469    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:23:27.391852 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.391826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:23:27.552236 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.552205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58j9r\" (UniqueName: \"kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r\") pod \"550b3445-4d15-44ad-bb94-eef6889630ee\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") "
Sep 09 21:23:27.552384 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.552245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls\") pod \"550b3445-4d15-44ad-bb94-eef6889630ee\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") "
Sep 09 21:23:27.552384 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.552321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config\") pod \"550b3445-4d15-44ad-bb94-eef6889630ee\" (UID: \"550b3445-4d15-44ad-bb94-eef6889630ee\") "
Sep 09 21:23:27.552689 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.552657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "550b3445-4d15-44ad-bb94-eef6889630ee" (UID: "550b3445-4d15-44ad-bb94-eef6889630ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:27.554382 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.554359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r" (OuterVolumeSpecName: "kube-api-access-58j9r") pod "550b3445-4d15-44ad-bb94-eef6889630ee" (UID: "550b3445-4d15-44ad-bb94-eef6889630ee"). InnerVolumeSpecName "kube-api-access-58j9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:27.554458 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.554383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "550b3445-4d15-44ad-bb94-eef6889630ee" (UID: "550b3445-4d15-44ad-bb94-eef6889630ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:27.653352 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.653326    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58j9r\" (UniqueName: \"kubernetes.io/projected/550b3445-4d15-44ad-bb94-eef6889630ee-kube-api-access-58j9r\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:27.653352 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.653350    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550b3445-4d15-44ad-bb94-eef6889630ee-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:27.653520 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.653359    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550b3445-4d15-44ad-bb94-eef6889630ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:27.698930 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.698902    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 09 21:23:27.703677 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.703646    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 21:23:27.839221 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.839194    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:23:27.955379 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.955315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls\") pod \"fadf395b-07b2-4e62-b0e1-083f927492db\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") "
Sep 09 21:23:27.955525 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.955390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2jm5\" (UniqueName: \"kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5\") pod \"fadf395b-07b2-4e62-b0e1-083f927492db\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") "
Sep 09 21:23:27.955525 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.955423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config\") pod \"fadf395b-07b2-4e62-b0e1-083f927492db\" (UID: \"fadf395b-07b2-4e62-b0e1-083f927492db\") "
Sep 09 21:23:27.955764 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.955727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fadf395b-07b2-4e62-b0e1-083f927492db" (UID: "fadf395b-07b2-4e62-b0e1-083f927492db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:27.957493 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.957466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fadf395b-07b2-4e62-b0e1-083f927492db" (UID: "fadf395b-07b2-4e62-b0e1-083f927492db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:27.957591 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:27.957511    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5" (OuterVolumeSpecName: "kube-api-access-b2jm5") pod "fadf395b-07b2-4e62-b0e1-083f927492db" (UID: "fadf395b-07b2-4e62-b0e1-083f927492db"). InnerVolumeSpecName "kube-api-access-b2jm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:28.054559 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.054530    2578 generic.go:356] "Generic (PLEG): container finished" podID="fadf395b-07b2-4e62-b0e1-083f927492db" containerID="9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105" exitCode=0
Sep 09 21:23:28.054999 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.054583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"
Sep 09 21:23:28.054999 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.054604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerDied","Data":"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"}
Sep 09 21:23:28.054999 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.054680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2" event={"ID":"fadf395b-07b2-4e62-b0e1-083f927492db","Type":"ContainerDied","Data":"807b1ae8472e5ac5f099bd116e5ecfc99999f71555ce1dee2fd1fe8d6beb23ed"}
Sep 09 21:23:28.054999 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.054715    2578 scope.go:117] "RemoveContainer" containerID="9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"
Sep 09 21:23:28.055842 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.055822    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fadf395b-07b2-4e62-b0e1-083f927492db-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:28.055926 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.055849    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2jm5\" (UniqueName: \"kubernetes.io/projected/fadf395b-07b2-4e62-b0e1-083f927492db-kube-api-access-b2jm5\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:28.055926 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.055864    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fadf395b-07b2-4e62-b0e1-083f927492db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:23:28.056391 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.056369    2578 generic.go:356] "Generic (PLEG): container finished" podID="550b3445-4d15-44ad-bb94-eef6889630ee" containerID="f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919" exitCode=0
Sep 09 21:23:28.056492 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.056444    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"
Sep 09 21:23:28.056558 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.056441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerDied","Data":"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"}
Sep 09 21:23:28.056617 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.056560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7" event={"ID":"550b3445-4d15-44ad-bb94-eef6889630ee","Type":"ContainerDied","Data":"d925d375dd93028409c9365324552f261b1f9520e5429e37b5b540f680c2d44d"}
Sep 09 21:23:28.056956 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.056933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:23:28.063071 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.063050    2578 scope.go:117] "RemoveContainer" containerID="9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"
Sep 09 21:23:28.070687 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.070674    2578 scope.go:117] "RemoveContainer" containerID="9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"
Sep 09 21:23:28.071031 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:28.071012    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd\": container with ID starting with 9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd not found: ID does not exist" containerID="9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"
Sep 09 21:23:28.071094 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.071038    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd"} err="failed to get container status \"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd\": rpc error: code = NotFound desc = could not find container \"9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd\": container with ID starting with 9e958efdf712270ad52abf4358f9e14321d8990df919288b6fe4ec223abfc9bd not found: ID does not exist"
Sep 09 21:23:28.071094 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.071063    2578 scope.go:117] "RemoveContainer" containerID="9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"
Sep 09 21:23:28.071343 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:28.071320    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105\": container with ID starting with 9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105 not found: ID does not exist" containerID="9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"
Sep 09 21:23:28.071395 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.071351    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105"} err="failed to get container status \"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105\": rpc error: code = NotFound desc = could not find container \"9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105\": container with ID starting with 9313ddbf012a6583fabd9546b773032f2cd35ee8d3916229e35abb44ab7b3105 not found: ID does not exist"
Sep 09 21:23:28.071395 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.071372    2578 scope.go:117] "RemoveContainer" containerID="6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"
Sep 09 21:23:28.079465 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.079445    2578 scope.go:117] "RemoveContainer" containerID="f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"
Sep 09 21:23:28.083460 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.083436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:23:28.086032 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.086005    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dc3b-predictor-778fbc896d-lwmr2"]
Sep 09 21:23:28.091482 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.091468    2578 scope.go:117] "RemoveContainer" containerID="6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"
Sep 09 21:23:28.091822 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:28.091805    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9\": container with ID starting with 6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9 not found: ID does not exist" containerID="6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"
Sep 09 21:23:28.091875 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.091829    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9"} err="failed to get container status \"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9\": rpc error: code = NotFound desc = could not find container \"6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9\": container with ID starting with 6287ef426186a65ddf405676f251982e4cf42d5c0f7bc2701711cfa27b20fdd9 not found: ID does not exist"
Sep 09 21:23:28.091875 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.091845    2578 scope.go:117] "RemoveContainer" containerID="f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"
Sep 09 21:23:28.092117 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:23:28.092098    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919\": container with ID starting with f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919 not found: ID does not exist" containerID="f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"
Sep 09 21:23:28.092171 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.092125    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919"} err="failed to get container status \"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919\": rpc error: code = NotFound desc = could not find container \"f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919\": container with ID starting with f78b20011041cc27a94549d3af727a50560f419c2fb61819c30d03a7c91c5919 not found: ID does not exist"
Sep 09 21:23:28.100388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.100368    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:23:28.104529 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:28.104512    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dc3b-predictor-56f6689897-dw9g7"]
Sep 09 21:23:29.285296 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:29.285258    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" path="/var/lib/kubelet/pods/550b3445-4d15-44ad-bb94-eef6889630ee/volumes"
Sep 09 21:23:29.285919 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:29.285895    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" path="/var/lib/kubelet/pods/fadf395b-07b2-4e62-b0e1-083f927492db/volumes"
Sep 09 21:23:32.048287 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:32.048258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:23:32.048726 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:32.048700    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:23:33.061169 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:33.061137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:23:33.061719 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:33.061692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:23:33.922828 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:33.922791    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 21:23:34.928818 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:34.928778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 09 21:23:42.049334 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:42.049289    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:23:43.061695 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:43.061649    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:23:43.922905 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:43.922876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:23:44.929758 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:44.929718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:23:52.049491 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:52.049445    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:23:53.062132 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:23:53.062090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:24:02.049408 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:02.049322    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 21:24:03.061895 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:03.061856    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 21:24:05.170921 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.170891    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:24:05.171326 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.171100    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container" containerID="cri-o://fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0" gracePeriod=30
Sep 09 21:24:05.171326 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.171139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kube-rbac-proxy" containerID="cri-o://7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29" gracePeriod=30
Sep 09 21:24:05.219195 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.219169    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:24:05.219415 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.219376    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container" containerID="cri-o://857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06" gracePeriod=30
Sep 09 21:24:05.219598 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.219498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kube-rbac-proxy" containerID="cri-o://6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b" gracePeriod=30
Sep 09 21:24:05.317067 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317039    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:24:05.317409 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317392    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317413    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container"
Sep 09 21:24:05.317457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317431    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container"
Sep 09 21:24:05.317457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317449    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container"
Sep 09 21:24:05.317457 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317455    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317462    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317466    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317525    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="550b3445-4d15-44ad-bb94-eef6889630ee" containerName="kserve-container"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kube-rbac-proxy"
Sep 09 21:24:05.317608 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.317546    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadf395b-07b2-4e62-b0e1-083f927492db" containerName="kserve-container"
Sep 09 21:24:05.320386 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.320372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.322941 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.322923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4085c-predictor-serving-cert\""
Sep 09 21:24:05.323049 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.322942    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4085c-kube-rbac-proxy-sar-config\""
Sep 09 21:24:05.333049 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.333028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:24:05.433129 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.433042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.433129 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.433080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsw7\" (UniqueName: \"kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.433314 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.433166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.436532 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.436505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:24:05.439925 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.439905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.442960 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.442937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4085c-predictor-serving-cert\""
Sep 09 21:24:05.443115 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.443098    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4085c-kube-rbac-proxy-sar-config\""
Sep 09 21:24:05.454234 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.454214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:24:05.533589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcg7l\" (UniqueName: \"kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.533734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.533734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.533734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsw7\" (UniqueName: \"kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.533937 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.533937 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.533860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.534362 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.534344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.536193 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.536173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.542516 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.542493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsw7\" (UniqueName: \"kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7\") pod \"success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") " pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.632484 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.632445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:05.634759 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.634725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.634863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.634801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcg7l\" (UniqueName: \"kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.634863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.634841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.634987 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:05.634975    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4085c-predictor-serving-cert: secret "error-404-isvc-4085c-predictor-serving-cert" not found
Sep 09 21:24:05.635054 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:05.635041    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls podName:bbd55279-9e31-44fc-a061-cde67e72e360 nodeName:}" failed. No retries permitted until 2026-09-09 21:24:06.135022648 +0000 UTC m=+1893.300349900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls") pod "error-404-isvc-4085c-predictor-77b774647b-8lm9b" (UID: "bbd55279-9e31-44fc-a061-cde67e72e360") : secret "error-404-isvc-4085c-predictor-serving-cert" not found
Sep 09 21:24:05.635392 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.635375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.651789 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.651769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcg7l\" (UniqueName: \"kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:05.916598 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:05.916566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:24:05.918582 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:24:05.918561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05eb807_4063_4b09_aae1_1de01a7cd17d.slice/crio-c517629bb97a5fdaf1338ee85de097ca614ab79aa6e49984bcb9012cb959f1be WatchSource:0}: Error finding container c517629bb97a5fdaf1338ee85de097ca614ab79aa6e49984bcb9012cb959f1be: Status 404 returned error can't find the container with id c517629bb97a5fdaf1338ee85de097ca614ab79aa6e49984bcb9012cb959f1be
Sep 09 21:24:06.139908 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.139876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:06.142276 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.142253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") pod \"error-404-isvc-4085c-predictor-77b774647b-8lm9b\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") " pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:06.231889 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.231819    2578 generic.go:356] "Generic (PLEG): container finished" podID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerID="6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b" exitCode=2
Sep 09 21:24:06.232264 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.231902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerDied","Data":"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"}
Sep 09 21:24:06.233731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.233712    2578 generic.go:356] "Generic (PLEG): container finished" podID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerID="7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29" exitCode=2
Sep 09 21:24:06.233841 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.233769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerDied","Data":"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"}
Sep 09 21:24:06.235428 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.235409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerStarted","Data":"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"}
Sep 09 21:24:06.235511 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.235464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerStarted","Data":"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"}
Sep 09 21:24:06.235511 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.235496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerStarted","Data":"c517629bb97a5fdaf1338ee85de097ca614ab79aa6e49984bcb9012cb959f1be"}
Sep 09 21:24:06.235612 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.235529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:06.257373 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.257333    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podStartSLOduration=1.257321477 podStartE2EDuration="1.257321477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:06.256525106 +0000 UTC m=+1893.421852369" watchObservedRunningTime="2026-09-09 21:24:06.257321477 +0000 UTC m=+1893.422648798"
Sep 09 21:24:06.349387 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.349353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:06.630766 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:06.630725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:24:06.632945 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:24:06.632911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd55279_9e31_44fc_a061_cde67e72e360.slice/crio-7a30324deeb19c21bdfa46e7d10a64f804050d4b71d9e3f0f914b30976c59973 WatchSource:0}: Error finding container 7a30324deeb19c21bdfa46e7d10a64f804050d4b71d9e3f0f914b30976c59973: Status 404 returned error can't find the container with id 7a30324deeb19c21bdfa46e7d10a64f804050d4b71d9e3f0f914b30976c59973
Sep 09 21:24:07.241918 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.241880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerStarted","Data":"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"}
Sep 09 21:24:07.242368 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.241981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerStarted","Data":"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"}
Sep 09 21:24:07.242368 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.242309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:07.242368 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.242356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerStarted","Data":"7a30324deeb19c21bdfa46e7d10a64f804050d4b71d9e3f0f914b30976c59973"}
Sep 09 21:24:07.242511 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.242382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:07.242511 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.242401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:07.243296 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.243275    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:07.243474 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.243454    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:07.264254 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:07.264218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podStartSLOduration=2.264206536 podStartE2EDuration="2.264206536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:07.262827398 +0000 UTC m=+1894.428154667" watchObservedRunningTime="2026-09-09 21:24:07.264206536 +0000 UTC m=+1894.429533804"
Sep 09 21:24:08.246795 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.246729    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:08.247190 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.246861    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:08.763912 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.763888    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:24:08.861429 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.861365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") pod \"18e55106-7ad0-415c-8298-1fc1b753b0ec\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") "
Sep 09 21:24:08.861429 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.861426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config\") pod \"18e55106-7ad0-415c-8298-1fc1b753b0ec\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") "
Sep 09 21:24:08.861632 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.861444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r6c7\" (UniqueName: \"kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7\") pod \"18e55106-7ad0-415c-8298-1fc1b753b0ec\" (UID: \"18e55106-7ad0-415c-8298-1fc1b753b0ec\") "
Sep 09 21:24:08.861839 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.861817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18e55106-7ad0-415c-8298-1fc1b753b0ec" (UID: "18e55106-7ad0-415c-8298-1fc1b753b0ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:24:08.863459 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.863433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18e55106-7ad0-415c-8298-1fc1b753b0ec" (UID: "18e55106-7ad0-415c-8298-1fc1b753b0ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:08.863569 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.863548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7" (OuterVolumeSpecName: "kube-api-access-2r6c7") pod "18e55106-7ad0-415c-8298-1fc1b753b0ec" (UID: "18e55106-7ad0-415c-8298-1fc1b753b0ec"). InnerVolumeSpecName "kube-api-access-2r6c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:08.962183 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.962147    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e55106-7ad0-415c-8298-1fc1b753b0ec-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:08.962183 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.962182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:08.962183 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:08.962193    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6c7\" (UniqueName: \"kubernetes.io/projected/18e55106-7ad0-415c-8298-1fc1b753b0ec-kube-api-access-2r6c7\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:09.251479 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.251454    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:24:09.252877 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.252853    2578 generic.go:356] "Generic (PLEG): container finished" podID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerID="857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06" exitCode=0
Sep 09 21:24:09.252968 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.252920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerDied","Data":"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"}
Sep 09 21:24:09.253013 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.252988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z" event={"ID":"47ac90c1-e488-44bd-ac79-a387359b50d4","Type":"ContainerDied","Data":"0239f0036e92980f2c644be1296f9201ad21af34332fe620d6c434be3df3c6f1"}
Sep 09 21:24:09.253065 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.253030    2578 scope.go:117] "RemoveContainer" containerID="6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"
Sep 09 21:24:09.254688 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.254666    2578 generic.go:356] "Generic (PLEG): container finished" podID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerID="fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0" exitCode=0
Sep 09 21:24:09.254863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.254709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerDied","Data":"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"}
Sep 09 21:24:09.254863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.254725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"
Sep 09 21:24:09.254863 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.254774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p" event={"ID":"18e55106-7ad0-415c-8298-1fc1b753b0ec","Type":"ContainerDied","Data":"99554b94af758ef5aa59379d0010da682ad355144675ea1e6d9c64be8c63095a"}
Sep 09 21:24:09.260786 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.260773    2578 scope.go:117] "RemoveContainer" containerID="857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"
Sep 09 21:24:09.264067 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.264049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config\") pod \"47ac90c1-e488-44bd-ac79-a387359b50d4\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") "
Sep 09 21:24:09.264151 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.264077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") pod \"47ac90c1-e488-44bd-ac79-a387359b50d4\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") "
Sep 09 21:24:09.264151 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.264118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44jcq\" (UniqueName: \"kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq\") pod \"47ac90c1-e488-44bd-ac79-a387359b50d4\" (UID: \"47ac90c1-e488-44bd-ac79-a387359b50d4\") "
Sep 09 21:24:09.264441 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.264412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47ac90c1-e488-44bd-ac79-a387359b50d4" (UID: "47ac90c1-e488-44bd-ac79-a387359b50d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:24:09.266241 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.266206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq" (OuterVolumeSpecName: "kube-api-access-44jcq") pod "47ac90c1-e488-44bd-ac79-a387359b50d4" (UID: "47ac90c1-e488-44bd-ac79-a387359b50d4"). InnerVolumeSpecName "kube-api-access-44jcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:09.266315 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.266272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47ac90c1-e488-44bd-ac79-a387359b50d4" (UID: "47ac90c1-e488-44bd-ac79-a387359b50d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:09.268333 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.268316    2578 scope.go:117] "RemoveContainer" containerID="6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"
Sep 09 21:24:09.268622 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:09.268606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b\": container with ID starting with 6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b not found: ID does not exist" containerID="6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"
Sep 09 21:24:09.268689 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.268633    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b"} err="failed to get container status \"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b\": rpc error: code = NotFound desc = could not find container \"6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b\": container with ID starting with 6ab532290fac70c556091b68e676e2e36762b6141438c23d8f8579d4f2ce055b not found: ID does not exist"
Sep 09 21:24:09.268689 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.268654    2578 scope.go:117] "RemoveContainer" containerID="857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"
Sep 09 21:24:09.268954 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:09.268933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06\": container with ID starting with 857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06 not found: ID does not exist" containerID="857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"
Sep 09 21:24:09.268997 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.268960    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06"} err="failed to get container status \"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06\": rpc error: code = NotFound desc = could not find container \"857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06\": container with ID starting with 857048632e6db039bb5277c1682579b6cd6001d21dc72d79385d45449c682a06 not found: ID does not exist"
Sep 09 21:24:09.268997 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.268974    2578 scope.go:117] "RemoveContainer" containerID="7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"
Sep 09 21:24:09.276326 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.276295    2578 scope.go:117] "RemoveContainer" containerID="fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"
Sep 09 21:24:09.290589 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.290517    2578 scope.go:117] "RemoveContainer" containerID="7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"
Sep 09 21:24:09.290878 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.290857    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:24:09.290971 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:09.290911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29\": container with ID starting with 7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29 not found: ID does not exist" containerID="7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"
Sep 09 21:24:09.290971 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.290944    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29"} err="failed to get container status \"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29\": rpc error: code = NotFound desc = could not find container \"7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29\": container with ID starting with 7ecdf0bb489d0fab5a98a394ac232762548d5cf55b40e4f8b933cb5d7f482a29 not found: ID does not exist"
Sep 09 21:24:09.290971 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.290961    2578 scope.go:117] "RemoveContainer" containerID="fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"
Sep 09 21:24:09.291257 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:24:09.291237    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0\": container with ID starting with fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0 not found: ID does not exist" containerID="fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"
Sep 09 21:24:09.291313 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.291267    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0"} err="failed to get container status \"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0\": rpc error: code = NotFound desc = could not find container \"fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0\": container with ID starting with fe7758c114f649e2771f6b2abda25cdded4124c9d181c402c67a88d7cc98ceb0 not found: ID does not exist"
Sep 09 21:24:09.300567 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.300547    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78161-predictor-6cfdb6f587-twf6p"]
Sep 09 21:24:09.365538 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.365512    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:09.365538 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.365535    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47ac90c1-e488-44bd-ac79-a387359b50d4-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:09.365689 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:09.365624    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jcq\" (UniqueName: \"kubernetes.io/projected/47ac90c1-e488-44bd-ac79-a387359b50d4-kube-api-access-44jcq\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:24:10.260051 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:10.260020    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"
Sep 09 21:24:10.278958 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:10.278925    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:24:10.285679 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:10.285650    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78161-predictor-74f5dcc9c4-lts9z"]
Sep 09 21:24:11.283833 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:11.283797    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" path="/var/lib/kubelet/pods/18e55106-7ad0-415c-8298-1fc1b753b0ec/volumes"
Sep 09 21:24:11.284436 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:11.284415    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" path="/var/lib/kubelet/pods/47ac90c1-e488-44bd-ac79-a387359b50d4/volumes"
Sep 09 21:24:12.049381 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:12.049356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:24:13.062799 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:13.062763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:24:13.251190 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:13.251165    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:13.251447 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:13.251428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:24:13.251609 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:13.251580    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:13.251870 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:13.251850    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:23.251972 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:23.251936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:23.252357 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:23.251936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:33.252224 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:33.252184    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:33.252224 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:33.252199    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:43.251786 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:43.251718    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 21:24:43.252278 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:43.251951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 21:24:53.252875 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:53.252835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:24:53.253300 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:24:53.252922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:33:20.135544 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.135507    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:33:20.138008 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.135794    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container" containerID="cri-o://f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77" gracePeriod=30
Sep 09 21:33:20.138008 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.135821    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kube-rbac-proxy" containerID="cri-o://7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39" gracePeriod=30
Sep 09 21:33:20.168707 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.168681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:33:20.168932 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.168906    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container" containerID="cri-o://6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6" gracePeriod=30
Sep 09 21:33:20.169004 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.168935    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kube-rbac-proxy" containerID="cri-o://3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe" gracePeriod=30
Sep 09 21:33:20.531170 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.531100    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbd55279-9e31-44fc-a061-cde67e72e360" containerID="3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe" exitCode=2
Sep 09 21:33:20.531316 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.531166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerDied","Data":"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"}
Sep 09 21:33:20.532922 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.532903    2578 generic.go:356] "Generic (PLEG): container finished" podID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerID="7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39" exitCode=2
Sep 09 21:33:20.533030 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:20.532938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerDied","Data":"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"}
Sep 09 21:33:23.026067 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.026039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:33:23.028872 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.028818    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:33:23.140156 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config\") pod \"bbd55279-9e31-44fc-a061-cde67e72e360\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") "
Sep 09 21:33:23.140293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") pod \"bbd55279-9e31-44fc-a061-cde67e72e360\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") "
Sep 09 21:33:23.140293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls\") pod \"d05eb807-4063-4b09-aae1-1de01a7cd17d\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") "
Sep 09 21:33:23.140293 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hsw7\" (UniqueName: \"kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7\") pod \"d05eb807-4063-4b09-aae1-1de01a7cd17d\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") "
Sep 09 21:33:23.140463 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcg7l\" (UniqueName: \"kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l\") pod \"bbd55279-9e31-44fc-a061-cde67e72e360\" (UID: \"bbd55279-9e31-44fc-a061-cde67e72e360\") "
Sep 09 21:33:23.140463 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config\") pod \"d05eb807-4063-4b09-aae1-1de01a7cd17d\" (UID: \"d05eb807-4063-4b09-aae1-1de01a7cd17d\") "
Sep 09 21:33:23.140590 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbd55279-9e31-44fc-a061-cde67e72e360" (UID: "bbd55279-9e31-44fc-a061-cde67e72e360"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:33:23.140718 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140693    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd55279-9e31-44fc-a061-cde67e72e360-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.141029 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.140819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d05eb807-4063-4b09-aae1-1de01a7cd17d" (UID: "d05eb807-4063-4b09-aae1-1de01a7cd17d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:33:23.142440 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.142403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbd55279-9e31-44fc-a061-cde67e72e360" (UID: "bbd55279-9e31-44fc-a061-cde67e72e360"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:33:23.142523 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.142445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d05eb807-4063-4b09-aae1-1de01a7cd17d" (UID: "d05eb807-4063-4b09-aae1-1de01a7cd17d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:33:23.142523 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.142482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l" (OuterVolumeSpecName: "kube-api-access-gcg7l") pod "bbd55279-9e31-44fc-a061-cde67e72e360" (UID: "bbd55279-9e31-44fc-a061-cde67e72e360"). InnerVolumeSpecName "kube-api-access-gcg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:33:23.142523 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.142486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7" (OuterVolumeSpecName: "kube-api-access-5hsw7") pod "d05eb807-4063-4b09-aae1-1de01a7cd17d" (UID: "d05eb807-4063-4b09-aae1-1de01a7cd17d"). InnerVolumeSpecName "kube-api-access-5hsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:33:23.241599 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.241574    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.241599 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.241597    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd55279-9e31-44fc-a061-cde67e72e360-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.241731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.241611    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05eb807-4063-4b09-aae1-1de01a7cd17d-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.241731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.241623    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsw7\" (UniqueName: \"kubernetes.io/projected/d05eb807-4063-4b09-aae1-1de01a7cd17d-kube-api-access-5hsw7\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.241731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.241635    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcg7l\" (UniqueName: \"kubernetes.io/projected/bbd55279-9e31-44fc-a061-cde67e72e360-kube-api-access-gcg7l\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:33:23.549459 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.549396    2578 generic.go:356] "Generic (PLEG): container finished" podID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerID="f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77" exitCode=0
Sep 09 21:33:23.549459 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.549447    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"
Sep 09 21:33:23.549656 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.549485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerDied","Data":"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"}
Sep 09 21:33:23.549656 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.549548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98" event={"ID":"d05eb807-4063-4b09-aae1-1de01a7cd17d","Type":"ContainerDied","Data":"c517629bb97a5fdaf1338ee85de097ca614ab79aa6e49984bcb9012cb959f1be"}
Sep 09 21:33:23.549656 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.549595    2578 scope.go:117] "RemoveContainer" containerID="7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"
Sep 09 21:33:23.551426 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.551388    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbd55279-9e31-44fc-a061-cde67e72e360" containerID="6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6" exitCode=0
Sep 09 21:33:23.551530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.551442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"
Sep 09 21:33:23.551530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.551479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerDied","Data":"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"}
Sep 09 21:33:23.551530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.551527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b" event={"ID":"bbd55279-9e31-44fc-a061-cde67e72e360","Type":"ContainerDied","Data":"7a30324deeb19c21bdfa46e7d10a64f804050d4b71d9e3f0f914b30976c59973"}
Sep 09 21:33:23.558053 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.558036    2578 scope.go:117] "RemoveContainer" containerID="f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"
Sep 09 21:33:23.565094 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.565066    2578 scope.go:117] "RemoveContainer" containerID="7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"
Sep 09 21:33:23.565330 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:33:23.565311    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39\": container with ID starting with 7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39 not found: ID does not exist" containerID="7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"
Sep 09 21:33:23.565378 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.565337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39"} err="failed to get container status \"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39\": rpc error: code = NotFound desc = could not find container \"7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39\": container with ID starting with 7336bea3e56080a7e2e2b34395d4e8b76e85f5453debb83f9fd2d2a939005a39 not found: ID does not exist"
Sep 09 21:33:23.565378 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.565352    2578 scope.go:117] "RemoveContainer" containerID="f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"
Sep 09 21:33:23.565607 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:33:23.565589    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77\": container with ID starting with f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77 not found: ID does not exist" containerID="f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"
Sep 09 21:33:23.565650 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.565612    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77"} err="failed to get container status \"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77\": rpc error: code = NotFound desc = could not find container \"f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77\": container with ID starting with f8f8d6a5214d826d560ea465ce7a55da985aa535ca3161d811ce41718a160d77 not found: ID does not exist"
Sep 09 21:33:23.565650 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.565625    2578 scope.go:117] "RemoveContainer" containerID="3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"
Sep 09 21:33:23.568672 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.568649    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:33:23.572706 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.572684    2578 scope.go:117] "RemoveContainer" containerID="6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"
Sep 09 21:33:23.577060 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.577041    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4085c-predictor-84bd9cf6c7-xrk98"]
Sep 09 21:33:23.580166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.580148    2578 scope.go:117] "RemoveContainer" containerID="3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"
Sep 09 21:33:23.580430 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:33:23.580414    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe\": container with ID starting with 3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe not found: ID does not exist" containerID="3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"
Sep 09 21:33:23.580523 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.580445    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe"} err="failed to get container status \"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe\": rpc error: code = NotFound desc = could not find container \"3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe\": container with ID starting with 3a2c4cdca2abe95533ee90651f88defa99bf62fd7245149426298ef6cf85ecbe not found: ID does not exist"
Sep 09 21:33:23.580523 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.580465    2578 scope.go:117] "RemoveContainer" containerID="6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"
Sep 09 21:33:23.580796 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:33:23.580775    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6\": container with ID starting with 6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6 not found: ID does not exist" containerID="6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"
Sep 09 21:33:23.580847 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.580802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6"} err="failed to get container status \"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6\": rpc error: code = NotFound desc = could not find container \"6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6\": container with ID starting with 6637a1f1d0c7f185b2b4ed9bd5261bb6b0d34fbb998ca97fe50142c972a052d6 not found: ID does not exist"
Sep 09 21:33:23.589274 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.589251    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:33:23.592193 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:23.592168    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4085c-predictor-77b774647b-8lm9b"]
Sep 09 21:33:25.282388 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:25.282356    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" path="/var/lib/kubelet/pods/bbd55279-9e31-44fc-a061-cde67e72e360/volumes"
Sep 09 21:33:25.282769 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:33:25.282737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" path="/var/lib/kubelet/pods/d05eb807-4063-4b09-aae1-1de01a7cd17d/volumes"
Sep 09 21:40:43.840943 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.840871    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:40:43.843233 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.841100    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container" containerID="cri-o://d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526" gracePeriod=30
Sep 09 21:40:43.843233 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.841222    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kube-rbac-proxy" containerID="cri-o://e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b" gracePeriod=30
Sep 09 21:40:43.947805 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.947774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:40:43.948012 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.947979    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container" containerID="cri-o://4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130" gracePeriod=30
Sep 09 21:40:43.948083 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:43.948027    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kube-rbac-proxy" containerID="cri-o://fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5" gracePeriod=30
Sep 09 21:40:44.304381 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:44.304310    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerID="fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5" exitCode=2
Sep 09 21:40:44.304568 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:44.304385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerDied","Data":"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"}
Sep 09 21:40:44.306191 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:44.306171    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerID="e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b" exitCode=2
Sep 09 21:40:44.306302 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:44.306244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerDied","Data":"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"}
Sep 09 21:40:46.682117 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.682096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:40:46.764499 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.764460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqzgk\" (UniqueName: \"kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk\") pod \"0c0a5fbe-bf23-44cb-a836-530405a30899\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") "
Sep 09 21:40:46.764619 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.764543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") pod \"0c0a5fbe-bf23-44cb-a836-530405a30899\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") "
Sep 09 21:40:46.764619 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.764565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config\") pod \"0c0a5fbe-bf23-44cb-a836-530405a30899\" (UID: \"0c0a5fbe-bf23-44cb-a836-530405a30899\") "
Sep 09 21:40:46.765059 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.764974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c0a5fbe-bf23-44cb-a836-530405a30899" (UID: "0c0a5fbe-bf23-44cb-a836-530405a30899"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:46.766464 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.766442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk" (OuterVolumeSpecName: "kube-api-access-kqzgk") pod "0c0a5fbe-bf23-44cb-a836-530405a30899" (UID: "0c0a5fbe-bf23-44cb-a836-530405a30899"). InnerVolumeSpecName "kube-api-access-kqzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:46.766561 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.766545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c0a5fbe-bf23-44cb-a836-530405a30899" (UID: "0c0a5fbe-bf23-44cb-a836-530405a30899"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:46.866090 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.866062    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzgk\" (UniqueName: \"kubernetes.io/projected/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-api-access-kqzgk\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:46.866090 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.866086    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0a5fbe-bf23-44cb-a836-530405a30899-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:46.866249 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.866096    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0a5fbe-bf23-44cb-a836-530405a30899-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:46.881243 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.881223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:40:46.967130 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.967106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config\") pod \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") "
Sep 09 21:40:46.967239 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.967139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") pod \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") "
Sep 09 21:40:46.967239 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.967202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scdjj\" (UniqueName: \"kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj\") pod \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\" (UID: \"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43\") "
Sep 09 21:40:46.967479 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.967455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" (UID: "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:40:46.969282 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.969260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" (UID: "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:46.969340 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:46.969300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj" (OuterVolumeSpecName: "kube-api-access-scdjj") pod "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" (UID: "3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43"). InnerVolumeSpecName "kube-api-access-scdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:47.068008 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.067965    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:47.068008 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.067983    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-proxy-tls\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:47.068008 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.067993    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdjj\" (UniqueName: \"kubernetes.io/projected/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43-kube-api-access-scdjj\") on node \"ip-10-0-142-161.ec2.internal\" DevicePath \"\""
Sep 09 21:40:47.319294 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.319274    2578 generic.go:356] "Generic (PLEG): container finished" podID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerID="4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130" exitCode=0
Sep 09 21:40:47.319408 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.319319    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"
Sep 09 21:40:47.319408 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.319358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerDied","Data":"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"}
Sep 09 21:40:47.319530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.319426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf" event={"ID":"3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43","Type":"ContainerDied","Data":"a10f18e8f78bcda2c666b91f1190c1b05d161d9547279907d190d10fdcafd19b"}
Sep 09 21:40:47.319530 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.319466    2578 scope.go:117] "RemoveContainer" containerID="fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"
Sep 09 21:40:47.321308 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.321285    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerID="d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526" exitCode=0
Sep 09 21:40:47.321422 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.321340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"
Sep 09 21:40:47.321422 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.321386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerDied","Data":"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"}
Sep 09 21:40:47.321548 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.321448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk" event={"ID":"0c0a5fbe-bf23-44cb-a836-530405a30899","Type":"ContainerDied","Data":"d8dd682fd3d01445bca10dd50626fbf10c59dabfda5a373f0df41e6abf8e8d7b"}
Sep 09 21:40:47.329799 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.329783    2578 scope.go:117] "RemoveContainer" containerID="4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"
Sep 09 21:40:47.340246 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.337953    2578 scope.go:117] "RemoveContainer" containerID="fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"
Sep 09 21:40:47.340389 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:40:47.340363    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5\": container with ID starting with fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5 not found: ID does not exist" containerID="fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"
Sep 09 21:40:47.340451 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.340400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5"} err="failed to get container status \"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5\": rpc error: code = NotFound desc = could not find container \"fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5\": container with ID starting with fb2793ff5129417687695fb09ce19d1577523b5db9f0b4ca41742b948cf4e5d5 not found: ID does not exist"
Sep 09 21:40:47.340451 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.340422    2578 scope.go:117] "RemoveContainer" containerID="4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"
Sep 09 21:40:47.340847 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:40:47.340829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130\": container with ID starting with 4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130 not found: ID does not exist" containerID="4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"
Sep 09 21:40:47.340925 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.340855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130"} err="failed to get container status \"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130\": rpc error: code = NotFound desc = could not find container \"4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130\": container with ID starting with 4766507e35698a263dfbf678a0013a782a3238fc29d5302086de9c8469beb130 not found: ID does not exist"
Sep 09 21:40:47.340925 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.340875    2578 scope.go:117] "RemoveContainer" containerID="e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"
Sep 09 21:40:47.347731 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.347709    2578 scope.go:117] "RemoveContainer" containerID="d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"
Sep 09 21:40:47.355239 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.355216    2578 scope.go:117] "RemoveContainer" containerID="e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"
Sep 09 21:40:47.355500 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:40:47.355480    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b\": container with ID starting with e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b not found: ID does not exist" containerID="e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"
Sep 09 21:40:47.355500 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.355494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:40:47.355602 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.355506    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b"} err="failed to get container status \"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b\": rpc error: code = NotFound desc = could not find container \"e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b\": container with ID starting with e6cb1f05cb9fb6f44976ef6e8cf1b8147ace40a1f8f2bb98ac48fe01e59ea88b not found: ID does not exist"
Sep 09 21:40:47.355602 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.355523    2578 scope.go:117] "RemoveContainer" containerID="d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"
Sep 09 21:40:47.355810 ip-10-0-142-161 kubenswrapper[2578]: E0909 21:40:47.355793    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526\": container with ID starting with d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526 not found: ID does not exist" containerID="d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"
Sep 09 21:40:47.355864 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.355814    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526"} err="failed to get container status \"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526\": rpc error: code = NotFound desc = could not find container \"d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526\": container with ID starting with d2f59ef176ca04c869f3d52d4fbe52a40ecb0ae0f788c90d7618159a368b0526 not found: ID does not exist"
Sep 09 21:40:47.360526 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.360500    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72d00-predictor-59d5cd5987-7hrxk"]
Sep 09 21:40:47.377418 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.377400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:40:47.391791 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:47.391771    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-72d00-predictor-54876486fd-sldbf"]
Sep 09 21:40:49.282515 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:49.282484    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" path="/var/lib/kubelet/pods/0c0a5fbe-bf23-44cb-a836-530405a30899/volumes"
Sep 09 21:40:49.282906 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:40:49.282893    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" path="/var/lib/kubelet/pods/3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43/volumes"
Sep 09 21:41:15.977548 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmz7m/must-gather-q465n"]
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977808    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977819    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977841    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977854    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977859    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977866    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977871    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977896    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977908    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977914    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977918    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977925    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977930    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977938    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977943    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977947    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977952    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977989    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.977996    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978002    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978006    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05eb807-4063-4b09-aae1-1de01a7cd17d" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978016    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd55279-9e31-44fc-a061-cde67e72e360" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978021    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978026    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e55106-7ad0-415c-8298-1fc1b753b0ec" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978031    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee610c2-dc5e-43b8-bf7b-1a67f6e63c43" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978036    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ac90c1-e488-44bd-ac79-a387359b50d4" containerName="kube-rbac-proxy"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978042    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kserve-container"
Sep 09 21:41:15.978026 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.978046    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0a5fbe-bf23-44cb-a836-530405a30899" containerName="kube-rbac-proxy"
Sep 09 21:41:15.980811 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.980795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:15.986816 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.986794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmz7m\"/\"kube-root-ca.crt\""
Sep 09 21:41:15.991334 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.991320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmz7m\"/\"openshift-service-ca.crt\""
Sep 09 21:41:15.992357 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:15.992343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmz7m\"/\"default-dockercfg-ss6ds\""
Sep 09 21:41:16.002194 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.002173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmz7m/must-gather-q465n"]
Sep 09 21:41:16.070542 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.070515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7fba452-7668-4494-b0d2-706a223bd824-must-gather-output\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.070648 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.070560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpzm\" (UniqueName: \"kubernetes.io/projected/e7fba452-7668-4494-b0d2-706a223bd824-kube-api-access-fgpzm\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.171148 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.171116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpzm\" (UniqueName: \"kubernetes.io/projected/e7fba452-7668-4494-b0d2-706a223bd824-kube-api-access-fgpzm\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.171272 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.171172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7fba452-7668-4494-b0d2-706a223bd824-must-gather-output\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.171444 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.171430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7fba452-7668-4494-b0d2-706a223bd824-must-gather-output\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.182357 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.182327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpzm\" (UniqueName: \"kubernetes.io/projected/e7fba452-7668-4494-b0d2-706a223bd824-kube-api-access-fgpzm\") pod \"must-gather-q465n\" (UID: \"e7fba452-7668-4494-b0d2-706a223bd824\") " pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.289207 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.289124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmz7m/must-gather-q465n"
Sep 09 21:41:16.572783 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.572757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmz7m/must-gather-q465n"]
Sep 09 21:41:16.574822 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:41:16.574795    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7fba452_7668_4494_b0d2_706a223bd824.slice/crio-8d432ff4f4d4b70f43d650466438c33bd4c0c21ce51e0195e7b88b2dc8a5d73b WatchSource:0}: Error finding container 8d432ff4f4d4b70f43d650466438c33bd4c0c21ce51e0195e7b88b2dc8a5d73b: Status 404 returned error can't find the container with id 8d432ff4f4d4b70f43d650466438c33bd4c0c21ce51e0195e7b88b2dc8a5d73b
Sep 09 21:41:16.576957 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:16.576944    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:41:17.441973 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:17.441936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmz7m/must-gather-q465n" event={"ID":"e7fba452-7668-4494-b0d2-706a223bd824","Type":"ContainerStarted","Data":"8d432ff4f4d4b70f43d650466438c33bd4c0c21ce51e0195e7b88b2dc8a5d73b"}
Sep 09 21:41:18.449969 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:18.449927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmz7m/must-gather-q465n" event={"ID":"e7fba452-7668-4494-b0d2-706a223bd824","Type":"ContainerStarted","Data":"e43ecb7aaa64abaac11563fc611f8f2147c36e97ff6552bbde2bdb8055b39e9d"}
Sep 09 21:41:18.450452 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:18.449983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmz7m/must-gather-q465n" event={"ID":"e7fba452-7668-4494-b0d2-706a223bd824","Type":"ContainerStarted","Data":"8273d18efcb848cf31c35e03dc691d343f136608c7858db82e4791e52e78b495"}
Sep 09 21:41:18.470406 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:18.470357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmz7m/must-gather-q465n" podStartSLOduration=2.5922676669999998 podStartE2EDuration="3.47034106s" totalImagesPullingTime="878.073393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:18.469034084 +0000 UTC m=+2925.634361353" watchObservedRunningTime="2026-09-09 21:41:18.47034106 +0000 UTC m=+2925.635668329"
Sep 09 21:41:18.990317 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:18.990289    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4wzrr_2d89900e-61d0-4ad4-9e29-8ffdc512f12b/global-pull-secret-syncer/0.log"
Sep 09 21:41:19.208574 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:19.208545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ldv7d_e1c247cc-2776-45c8-9fae-f0374394b7a7/konnectivity-agent/0.log"
Sep 09 21:41:19.370869 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:19.370835    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-161.ec2.internal_3a96d69af6916da553cbc5c637b97bef/haproxy/0.log"
Sep 09 21:41:23.387087 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:23.387015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78588c9594-6hwfm_b00090f2-28d4-4b54-a9a1-4f3c24094d99/metrics-server/0.log"
Sep 09 21:41:23.588977 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:23.588946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59lhg_087a2bcc-a7cd-4a95-b913-9318847cfb5d/node-exporter/0.log"
Sep 09 21:41:23.658181 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:23.658107    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59lhg_087a2bcc-a7cd-4a95-b913-9318847cfb5d/kube-rbac-proxy/0.log"
Sep 09 21:41:23.700462 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:23.700364    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59lhg_087a2bcc-a7cd-4a95-b913-9318847cfb5d/init-textfile/0.log"
Sep 09 21:41:24.577871 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.577821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/thanos-query/0.log"
Sep 09 21:41:24.629844 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.629809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/kube-rbac-proxy-web/0.log"
Sep 09 21:41:24.677990 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.677845    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/kube-rbac-proxy/0.log"
Sep 09 21:41:24.719054 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.719016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/prom-label-proxy/0.log"
Sep 09 21:41:24.777577 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.777534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/kube-rbac-proxy-rules/0.log"
Sep 09 21:41:24.815529 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:24.815400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb486c4bf-hg6dx_8b88e86f-c45e-42d9-a9c3-52a8fbc964e0/kube-rbac-proxy-metrics/0.log"
Sep 09 21:41:26.947218 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:26.947186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"]
Sep 09 21:41:26.952433 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:26.952406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:26.972515 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:26.972483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"]
Sep 09 21:41:27.064734 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.064686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c764c7d8d-5s7g2_50599695-bf36-40ab-abb1-b490abbd407c/console/0.log"
Sep 09 21:41:27.087219 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.087188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-proc\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.087219 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.087221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grt9f\" (UniqueName: \"kubernetes.io/projected/f7751768-ef0f-4505-bb87-e6d974e000a1-kube-api-access-grt9f\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.087412 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.087262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-podres\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.087412 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.087290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-sys\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.087412 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.087304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-lib-modules\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.110674 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.110640    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-zgc8d_9eeccdf2-6b0f-4c60-96b1-dbdb97e819d8/download-server/0.log"
Sep 09 21:41:27.188104 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-proc\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188104 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grt9f\" (UniqueName: \"kubernetes.io/projected/f7751768-ef0f-4505-bb87-e6d974e000a1-kube-api-access-grt9f\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-podres\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-proc\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-podres\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-sys\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-lib-modules\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188356 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-sys\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.188683 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.188436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7751768-ef0f-4505-bb87-e6d974e000a1-lib-modules\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.196724 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.196694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grt9f\" (UniqueName: \"kubernetes.io/projected/f7751768-ef0f-4505-bb87-e6d974e000a1-kube-api-access-grt9f\") pod \"perf-node-gather-daemonset-jxlt9\" (UID: \"f7751768-ef0f-4505-bb87-e6d974e000a1\") " pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.265060 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.264983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:27.546677 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:27.546643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"]
Sep 09 21:41:27.549915 ip-10-0-142-161 kubenswrapper[2578]: W0909 21:41:27.549876    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7751768_ef0f_4505_bb87_e6d974e000a1.slice/crio-a1aa56731965c9b6a0df1b42a4c6365d1d4832b87402e4096a2200a1037d5d86 WatchSource:0}: Error finding container a1aa56731965c9b6a0df1b42a4c6365d1d4832b87402e4096a2200a1037d5d86: Status 404 returned error can't find the container with id a1aa56731965c9b6a0df1b42a4c6365d1d4832b87402e4096a2200a1037d5d86
Sep 09 21:41:28.509056 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.509020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9" event={"ID":"f7751768-ef0f-4505-bb87-e6d974e000a1","Type":"ContainerStarted","Data":"efd48c4194bd18cf46e415ff04d088a1cc80a2b2ac0025e7a34ea54129c41393"}
Sep 09 21:41:28.509520 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.509072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9" event={"ID":"f7751768-ef0f-4505-bb87-e6d974e000a1","Type":"ContainerStarted","Data":"a1aa56731965c9b6a0df1b42a4c6365d1d4832b87402e4096a2200a1037d5d86"}
Sep 09 21:41:28.509520 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.509158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:28.526255 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.526193    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9" podStartSLOduration=2.52615931 podStartE2EDuration="2.52615931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:41:28.524734214 +0000 UTC m=+2935.690061482" watchObservedRunningTime="2026-09-09 21:41:28.52615931 +0000 UTC m=+2935.691486579"
Sep 09 21:41:28.721833 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.721768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5d2_d617a952-4648-4c17-9447-69f97fbbe210/dns/0.log"
Sep 09 21:41:28.744160 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.744133    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5d2_d617a952-4648-4c17-9447-69f97fbbe210/kube-rbac-proxy/0.log"
Sep 09 21:41:28.828276 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:28.828250    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qdng8_56d46fb7-8c1e-42b8-9dd8-42cf2b2f70d8/dns-node-resolver/0.log"
Sep 09 21:41:29.380694 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:29.380665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c7cjx_02664476-02ce-4f9e-9309-b18a425e76c6/node-ca/0.log"
Sep 09 21:41:30.507445 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:30.507417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qt4t_69efb2bd-e552-4e15-8b96-d79aad2358a5/serve-healthcheck-canary/0.log"
Sep 09 21:41:31.028370 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:31.028336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk2zv_e3bf0307-f669-46a2-b79f-a7038ffe5b74/kube-rbac-proxy/0.log"
Sep 09 21:41:31.049535 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:31.049507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk2zv_e3bf0307-f669-46a2-b79f-a7038ffe5b74/exporter/0.log"
Sep 09 21:41:31.072283 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:31.072255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk2zv_e3bf0307-f669-46a2-b79f-a7038ffe5b74/extractor/0.log"
Sep 09 21:41:33.449873 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:33.449838    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-qj2kw_816429ee-14e0-4fa0-8506-d999ffc0ac26/manager/0.log"
Sep 09 21:41:34.523115 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:34.523083    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmz7m/perf-node-gather-daemonset-jxlt9"
Sep 09 21:41:39.607137 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.607109    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:41:39.628548 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.628527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/egress-router-binary-copy/0.log"
Sep 09 21:41:39.650406 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.650380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/cni-plugins/0.log"
Sep 09 21:41:39.672166 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.672139    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/bond-cni-plugin/0.log"
Sep 09 21:41:39.693188 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.693156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/routeoverride-cni/0.log"
Sep 09 21:41:39.713812 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.713733    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/whereabouts-cni-bincopy/0.log"
Sep 09 21:41:39.734827 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.734806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vfjds_84aa425b-634f-42cb-8758-a6980a14f99e/whereabouts-cni/0.log"
Sep 09 21:41:39.810093 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.810058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v2qvz_0861aad9-818e-46fb-bef9-06e1d0d99c6b/kube-multus/0.log"
Sep 09 21:41:39.864419 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.864345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5k56s_ee1b4883-3bbb-459e-b700-5a38c2131ba3/network-metrics-daemon/0.log"
Sep 09 21:41:39.883767 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:39.883724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5k56s_ee1b4883-3bbb-459e-b700-5a38c2131ba3/kube-rbac-proxy/0.log"
Sep 09 21:41:40.984198 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:40.984168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/ovn-controller/0.log"
Sep 09 21:41:41.000949 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.000922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/ovn-acl-logging/0.log"
Sep 09 21:41:41.023913 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.023889    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/kube-rbac-proxy-node/0.log"
Sep 09 21:41:41.044352 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.044330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:41:41.064982 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.064964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/northd/0.log"
Sep 09 21:41:41.086898 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.086877    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/nbdb/0.log"
Sep 09 21:41:41.108454 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.108431    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/sbdb/0.log"
Sep 09 21:41:41.152205 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:41.152182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dv7rn_2ce78ece-1667-4ec6-be73-9f59da6e3b72/ovnkube-controller/0.log"
Sep 09 21:41:42.383181 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:42.383155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9g5pw_14fd5f9d-1f7d-4310-9326-06bb7c2c9bd5/network-check-target-container/0.log"
Sep 09 21:41:43.435508 ip-10-0-142-161 kubenswrapper[2578]: I0909 21:41:43.435478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g64bs_41323f0f-2da9-4852-bb4a-92ea2f770f87/iptables-alerter/0.log"