Sep 16 18:42:21.073597 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:42:21.073612 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:42:21.073621 ip-10-0-128-253 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:42:21.073801 ip-10-0-128-253 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:42:31.107660 ip-10-0-128-253 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:42:31.107677 ip-10-0-128-253 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 516e26f8084944c79601cbc92c479ae7 --
Sep 16 18:44:29.152258 ip-10-0-128-253 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:29.519788 ip-10-0-128-253 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.519788 ip-10-0-128-253 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:29.519788 ip-10-0-128-253 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.519788 ip-10-0-128-253 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:29.519788 ip-10-0-128-253 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.522195 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.522108    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:29.527546 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527532    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.527546 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527547    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527552    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527555    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527558    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527561    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527564    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527566    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527569    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527573    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527577    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527579    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527583    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527585    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527588    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527591    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527594    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527597    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527605    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527608    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.527605 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527611    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527614    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527618    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527621    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527623    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527626    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527628    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527631    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527633    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527636    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527638    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527641    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527643    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527646    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527648    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527651    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527653    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527656    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527658    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527660    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.528070 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527664    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527667    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527670    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527672    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527675    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527677    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527680    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527683    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527686    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527689    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527692    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527695    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527698    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527701    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527705    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527708    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527711    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527713    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527715    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.528562 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527718    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527720    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527723    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527725    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527728    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527730    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527733    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527735    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527738    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527740    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527743    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527745    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527748    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527750    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527753    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527755    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527758    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527760    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527762    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527766    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.529039 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527768    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527771    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527773    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527776    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527778    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527781    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527784    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527787    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527789    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527791    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527794    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527797    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527799    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527801    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527803    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527806    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527808    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527811    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527814    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.527816    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.529524 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529305    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529320    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529324    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529327    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529330    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529333    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529335    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529339    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529341    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529344    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529348    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529351    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529355    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529358    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529360    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529363    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529366    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529368    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529372    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529376    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.530050 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529379    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529381    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529384    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529386    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529389    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529392    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529395    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529397    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529399    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529402    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529404    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529407    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529409    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529412    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529414    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529417    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529419    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529422    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529424    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.530551 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529427    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529429    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529431    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529434    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529436    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529438    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529441    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529443    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529445    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529448    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529450    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529453    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529456    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529460    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529462    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529465    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529467    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529470    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529472    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529475    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.531024 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529477    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529479    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529482    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529484    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529487    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529489    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529491    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529494    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529496    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529498    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529501    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529504    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529506    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529508    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529511    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529513    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529516    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529518    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529520    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.531511 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529523    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529525    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529528    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529530    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529533    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529539    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529542    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529545    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529548    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529551    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529553    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529556    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529558    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529561    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529563    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529567    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529570    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529573    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529576    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.531972 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529578    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529581    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529961    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529965    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529968    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529971    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529973    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529976    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529979    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529996    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.529998    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530001    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530004    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530006    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530008    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530011    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530013    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530016    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530019    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530022    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.532452 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530025    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530028    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530030    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530033    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530035    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530038    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530040    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530042    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530046    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530050    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530053    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530056    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530058    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530061    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530063    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530066    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530068    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530071    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530074    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.532932 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530076    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530079    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530083    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530086    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530089    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530092    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530094    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530097    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530100    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530102    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530105    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530108    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530110    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530113    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530115    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530118    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530120    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530123    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530126    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.533414 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530128    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530131    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530134    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530136    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530139    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530143    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530146    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530150    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530152    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530155    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530157    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530162    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530165    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530168    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530170    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530173    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530175    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530178    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.533882 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530180    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530182    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530185    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530187    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530190    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530192    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530194    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530197    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530199    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530202    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530204    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530206    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530209    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530211    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530213    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530216    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530218    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530220    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530223    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530225    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.534361 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530227    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530230    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530232    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530234    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530299    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530336    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530342    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530346    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530349    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530353    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530356    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530360    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530363    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530366    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530370    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530373    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530379    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530382    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530385    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530388    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530391    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530394    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530398    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530401    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:29.534888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530404    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530408    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530411    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530414    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530417    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530420    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530423    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530426    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530429    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530432    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530435    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530437    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530444    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530447    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530450    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530453    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530456    2580 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530458    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530462    2580 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530465    2580 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530468    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530470    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530473    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530477    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:29.535591 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530481    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530484    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530487    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530490    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530492    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530495    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530498    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530501    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530504    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530507    2580 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530511    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530513    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530516    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530519    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530522    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530525    2580 flags.go:64] FLAG: --help="false"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530528    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530531    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530534    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530536    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530542    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530545    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530548    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:29.536197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530551    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530554    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530557    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530560    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530563    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530566    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530569    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530571    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530574    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530578    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530581    2580 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530584    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530587    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530590    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530595    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530598    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530601    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530604    2580 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530606    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530610    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530612    2580 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530615    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530619    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530622    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530626    2580 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:29.536767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530628    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530631    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530634    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530637    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530641    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530644    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530647    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530654    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530657    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530660    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530663    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530666    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530671    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530674    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530677    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530680    2580 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530683    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530687    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce215dcb1325b6ea"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530691    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530693    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530696    2580 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530699    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530702    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530705    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:29.537428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530708    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530711    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530714    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530717    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530720    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530723    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530725    2580 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530728    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530731    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530734    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530737    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530739    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530744    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530747    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530750    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530752    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530755    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530758    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530761    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530764    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530767    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530770    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530775    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530778    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530780    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:29.538040 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530788    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530790    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530793    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530796    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530799    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530801    2580 flags.go:64] FLAG: --v="2"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530805    2580 flags.go:64] FLAG: --version="false"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530809    2580 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530813    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.530816    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530933    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530937    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530939    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530942    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530944    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530947    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530949    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530952    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530954    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530958    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530960    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530962    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.538686 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530965    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530967    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530970    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530972    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530974    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530977    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530979    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.530998    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531002    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531004    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531008    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531011    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531013    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531016    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531018    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531021    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531023    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531026    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531028    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.539363 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531031    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531034    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531036    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531039    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531041    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531044    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531047    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531049    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531052    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531054    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531059    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531061    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531064    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531066    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531069    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531071    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531076    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531081    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531085    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531090    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.539998 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531093    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531095    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531098    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531103    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531106    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531108    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531111    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531114    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531116    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531119    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531121    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531124    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531126    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531129    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531132    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531135    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531138    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531140    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531143    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531145    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.540730 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531148    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531150    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531154    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531157    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531159    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531162    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531164    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531166    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531169    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531171    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531174    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531176    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531179    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531181    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531183    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531187    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531190    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531192    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531195    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.541382 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531198    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531202    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531205    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531207    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531210    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531213    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531216    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531219    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.531221    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.531711    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.541413    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.541428    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.541447    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.541453    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541521    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.541852 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541527    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541530    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541533    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541536    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541539    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541541    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541544    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541547    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541550    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541553    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541555    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541558    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541561    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541563    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541566    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541569    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541572    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541574    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541577    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541579    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.542343 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541582    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541585    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541587    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541590    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541594    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541598    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541601    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541604    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541607    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541609    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541612    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541614    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541617    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541620    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541622    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541625    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541627    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541630    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541632    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.542853 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541635    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541637    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541639    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541642    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541645    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541647    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541650    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541653    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541656    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541659    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541661    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541664    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541666    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541669    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541671    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541674    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541676    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541679    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541681    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541684    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.543331 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541686    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541689    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541691    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541694    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541697    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541699    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541702    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541705    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541708    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541710    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541713    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541715    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541718    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541720    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541723    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541725    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541728    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541730    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541733    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541737    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.543807 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541740    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541742    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541745    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541747    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541750    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541752    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541756    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541760    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541762    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541765    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541768    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541772    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541775    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541778    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541782    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541785    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541787    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541790    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.544307 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541793    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.541798    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541916    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541921    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541924    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541927    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541931    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541933    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541936    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541939    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541942    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541947    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541950    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541953    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.544740 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541956    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541959    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541961    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541964    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541967    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541969    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541972    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541974    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541977    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.541979    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542001    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542005    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542007    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542010    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542013    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542016    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542019    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542021    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542024    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.545102 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542026    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542029    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542031    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542034    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542036    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542040    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542043    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542046    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542049    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542051    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542054    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542056    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542059    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542061    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542065    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542067    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542070    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542072    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542074    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542077    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.545569 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542079    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542082    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542084    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542086    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542089    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542091    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542094    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542096    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542099    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542102    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542104    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542107    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542110    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542112    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542115    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542117    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542119    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542122    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542125    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542127    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.546062 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542130    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542132    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542134    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542137    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542139    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542142    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542144    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542147    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542149    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542152    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542154    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542157    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542159    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542180    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542183    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542186    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542189    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542192    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542194    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542197    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.546534 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542201    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542205    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542208    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542211    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542214    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542216    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542219    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:29.542222    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.542226    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.542321    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.544865    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.545833    2580 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.545927    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:29.547038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.545961    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:29.548510 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.548497    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:29.551027 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.551009    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:29.558296 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.558279    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:29.562773 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.562755    2580 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:29.563650 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.563635    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:29.569110 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.569091    2580 fs.go:135] Filesystem UUIDs: map[73df6aaf-fc3a-48fc-8341-bd521ac0f86f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1cf36a6-b7c7-4300-9c3b-6ad9af364688:/dev/nvme0n1p3]
Sep 16 18:44:29.569184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.569108    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:29.575244 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.575226    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:29.575378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.575244    2580 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:29.573944874 +0000 UTC m=+0.319440669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094640 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219a04d00552b80f5006733cc88863 SystemUUID:ec219a04-d005-52b8-0f50-06733cc88863 BootID:516e26f8-0849-44c7-9601-cbc92c479ae7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:c9:ca:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:c9:ca:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:de:15:06:77:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:29.575378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.575375    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:29.575470 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.575452    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:29.577119 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.577099    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:29.577278 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.577122    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:29.577778 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.577768    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:29.577808 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.577781    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:29.577808 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.577788    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:29.578271 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.578261    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:29.579122 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.579113    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:29.583059 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583048    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:29.583185 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583176    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:29.583217 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583205    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:29.583244 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583222    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:29.583244 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583236    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:29.583346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583262    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:29.583346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583273    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:29.583903 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583868    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:29.584007 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.583907    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:29.586814 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.586745    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:29.588541 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.588523    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:29.588616 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.588555    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:29.590087 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590076    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:29.590133 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590095    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:29.590133 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590103    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:29.590133 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590109    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:29.590133 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590123    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:29.590133 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590132    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590153    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590159    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590168    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590175    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590186    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:29.590261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.590191    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:29.591342 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.591333    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:29.591342 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.591343    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:29.594361 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.594328    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:29.594438 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.594425    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:29.594493 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.594464    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:29.595291 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.595279    2580 server.go:1268] "Started kubelet"
Sep 16 18:44:29.595409 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.595385    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:29.595476 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.595430    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:29.595525 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.595516    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:29.596524 ip-10-0-128-253 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:29.598535 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.598369    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:29.599533 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.599515    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:29.599972 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.599951    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:29.602549 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.601656    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-253.ec2.internal.18d5e1695cda1d08  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-253.ec2.internal,UID:ip-10-0-128-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-253.ec2.internal,},FirstTimestamp:2026-09-16 18:44:29.595254024 +0000 UTC m=+0.340749813,LastTimestamp:2026-09-16 18:44:29.595254024 +0000 UTC m=+0.340749813,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-253.ec2.internal,}"
Sep 16 18:44:29.603423 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.603408    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:29.603971 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.603954    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:29.604180 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.604159    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:29.604741 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.604723    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:29.604741 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.604726    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:29.604853 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.604750    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:29.605103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.604976    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:29.605103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605082    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:29.605547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605527    2580 factory.go:55] Registering systemd factory
Sep 16 18:44:29.605547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605541    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:29.605748 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605733    2580 factory.go:153] Registering CRI-O factory
Sep 16 18:44:29.605798 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605751    2580 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:29.605798 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605794    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:29.605862 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605809    2580 factory.go:103] Registering Raw factory
Sep 16 18:44:29.605862 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.605818    2580 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:29.606775 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.606756    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:44:29.606951 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.606940    2580 manager.go:349] Starting recovery of all containers
Sep 16 18:44:29.607173 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.607147    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:44:29.612130 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.612107    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8xbb6"
Sep 16 18:44:29.618650 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.618631    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8xbb6"
Sep 16 18:44:29.620333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.620321    2580 manager.go:354] Recovery completed
Sep 16 18:44:29.621634 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.621612    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:44:29.624441 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.624430    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.626050 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626035    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.626106 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626062    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.626106 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626072    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.626655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626640    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:29.626655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626655    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:29.626776 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626662    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.626776 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.626672    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:29.627329 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.627311    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.627377 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.627355    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.627377 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.627369    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.628067 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.627979    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-253.ec2.internal.18d5e1695eb0039b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-253.ec2.internal,UID:ip-10-0-128-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-253.ec2.internal,},FirstTimestamp:2026-09-16 18:44:29.626049435 +0000 UTC m=+0.371545222,LastTimestamp:2026-09-16 18:44:29.626049435 +0000 UTC m=+0.371545222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-253.ec2.internal,}"
Sep 16 18:44:29.628610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.628598    2580 policy_none.go:49] "None policy: Start"
Sep 16 18:44:29.628673 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.628615    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:29.628673 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.628624    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:29.630531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.630520    2580 policy_none.go:47] "Start"
Sep 16 18:44:29.665623 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.665606    2580 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.665649    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.665659    2580 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.666007    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.666022    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.666108    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.666181    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.666189    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.666649    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:29.689710 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.666697    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:29.737224 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.737189    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:29.738593 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.738563    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:29.738687 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.738634    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:29.738687 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.738663    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:29.738775 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.738714    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:29.740833 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.740818    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:29.766780 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.766765    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.767585 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.767571    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.767677 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.767603    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.767677 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.767618    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.767677 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.767650    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.788812 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.788745    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.788812 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.788766    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-253.ec2.internal\": node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:29.826754 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.826736    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:29.839059 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.839039    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"]
Sep 16 18:44:29.839115 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.839101    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.839935 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.839917    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.840025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.839950    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.840025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.839965    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.842247 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842233    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.842397 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.842437 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842400    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.842898 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842881    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.842961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842904    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.842961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842915    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.842961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842944    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.842961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842961    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.843169 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.842971    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.845162 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.845148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.845209 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.845182    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.845837 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.845820    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.845914 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.845855    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.845914 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.845866    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.861515 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.861496    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-253.ec2.internal\" not found" node="ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.864911 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.864895    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-253.ec2.internal\" not found" node="ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.907156 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.907130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.907256 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.907158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.907256 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:29.907175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a2fa8135ce3bbe90bb18a946454dc57-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"6a2fa8135ce3bbe90bb18a946454dc57\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:29.927388 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:29.927356    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.008086 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.008196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.008196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a2fa8135ce3bbe90bb18a946454dc57-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"6a2fa8135ce3bbe90bb18a946454dc57\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.008196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a2fa8135ce3bbe90bb18a946454dc57-config\") pod \"kube-apiserver-proxy-ip-10-0-128-253.ec2.internal\" (UID: \"6a2fa8135ce3bbe90bb18a946454dc57\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.008196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.008196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.008161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd7072b9c9593b3032f24b69cb3ef8c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal\" (UID: \"cd7072b9c9593b3032f24b69cb3ef8c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.028172 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.028151    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.129038 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.128950    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.164122 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.164101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.167469 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.167453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:30.229531 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.229505    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.330051 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.330029    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.430560 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.430486    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.530993 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.530971    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.546137 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.546116    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:30.546262 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.546247    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:30.625232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.625203    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:30 +0000 UTC" deadline="2028-06-11 16:57:06.552329006 +0000 UTC"
Sep 16 18:44:30.625232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.625230    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15214h12m35.927102093s"
Sep 16 18:44:30.631392 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.631367    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.686215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.686159    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:30.732130 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.732103    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.769906 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.769885    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:30.774119 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:30.774091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2fa8135ce3bbe90bb18a946454dc57.slice/crio-7c5314e8d9ba5545cc2ef4c8e263010c697498db8c268a5d4bc08687a69c5597 WatchSource:0}: Error finding container 7c5314e8d9ba5545cc2ef4c8e263010c697498db8c268a5d4bc08687a69c5597: Status 404 returned error can't find the container with id 7c5314e8d9ba5545cc2ef4c8e263010c697498db8c268a5d4bc08687a69c5597
Sep 16 18:44:30.774466 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:30.774446    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7072b9c9593b3032f24b69cb3ef8c2.slice/crio-a4cab636f01bd4737bfed14edbdf7099c32bb18def94dfbb475bafea8b2e3c84 WatchSource:0}: Error finding container a4cab636f01bd4737bfed14edbdf7099c32bb18def94dfbb475bafea8b2e3c84: Status 404 returned error can't find the container with id a4cab636f01bd4737bfed14edbdf7099c32bb18def94dfbb475bafea8b2e3c84
Sep 16 18:44:30.779796 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:30.779778    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:30.833182 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.833155    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:30.934434 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:30.934402    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:31.035357 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.035281    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-253.ec2.internal\" not found"
Sep 16 18:44:31.036225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.036208    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:31.105419 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.105381    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:31.132317 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.132286    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:31.133289 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.133271    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal"
Sep 16 18:44:31.142465 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.142440    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:31.585183 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.585155    2580 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:31.593968 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.593784    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:31.594270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.594240    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w","openshift-cluster-node-tuning-operator/tuned-fxkql","openshift-dns/node-resolver-bf6tv","openshift-multus/multus-additional-cni-plugins-cjsrm","openshift-multus/multus-ghj9s","openshift-multus/network-metrics-daemon-svtlg","kube-system/konnectivity-agent-ftd65","openshift-image-registry/node-ca-zfpxr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal","openshift-network-diagnostics/network-check-target-h44vj","openshift-network-operator/iptables-alerter-m977x","openshift-ovn-kubernetes/ovnkube-node-pqmkz","kube-system/global-pull-secret-syncer-6nq5t"]
Sep 16 18:44:31.597004 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.596978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.599391 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.599365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.599897 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.599877    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l7gc7\""
Sep 16 18:44:31.599994 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.599931    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:31.599994 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.599983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:31.601856 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.601836    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.602130 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.602112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.602642 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.602250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.602642 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.602301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:31.602642 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.602458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dlxff\""
Sep 16 18:44:31.603859 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.603835    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:31.604821 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.604800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.605766 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.605745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.606290 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.606270    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:31.606361 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.606319    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:31.606677 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.606651    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fdsp\""
Sep 16 18:44:31.606790 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.606771    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:31.606900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.606881    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.607054 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.607037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:31.607322 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.607303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.609478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.609460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbxtx\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611413    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tlqbc\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611544    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.611698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.611613    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:31.612153 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.612136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.612468 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.612444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:31.612610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.612591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mjbtv\""
Sep 16 18:44:31.612712 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.612688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.612877 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.612859    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:31.613181 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.613162    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.613305 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.612270    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:31.615216 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.615182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.618532 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-kubelet\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618634 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-device-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.618634 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.618752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-etc-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-registration-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.618752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1506a9e-c134-411b-8a43-0ff0bd857d2c-host\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.618752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-systemd-units\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-sys-fs\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-slash\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-netd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-script-lib\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-agent-certs\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-var-lib-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.618922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-log-socket\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6cb\" (UniqueName: \"kubernetes.io/projected/39723fd9-72ef-45a2-8c31-54d4384ee595-kube-api-access-xp6cb\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.618975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-config\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-systemd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-env-overrides\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39723fd9-72ef-45a2-8c31-54d4384ee595-ovn-node-metrics-cert\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1506a9e-c134-411b-8a43-0ff0bd857d2c-serviceca\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-socket-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.619245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f7fa222-cc1f-402d-bd83-cc799812dfc0-iptables-alerter-script\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9mb\" (UniqueName: \"kubernetes.io/projected/e1506a9e-c134-411b-8a43-0ff0bd857d2c-kube-api-access-zf9mb\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-konnectivity-ca\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-node-log\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7sc\" (UniqueName: \"kubernetes.io/projected/d7a2e49a-5573-433a-aab1-84a96e43720a-kube-api-access-8r7sc\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-netns\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-ovn\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-bin\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.619546 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f7fa222-cc1f-402d-bd83-cc799812dfc0-host-slash\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.619860 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sln\" (UniqueName: \"kubernetes.io/projected/5f7fa222-cc1f-402d-bd83-cc799812dfc0-kube-api-access-n4sln\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.619860 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.619614    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:31.620471 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.620447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.622281 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.622255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.622588 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.622570    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.622973 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.622782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcrfn\""
Sep 16 18:44:31.623242 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.623222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:31.623335 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.623300    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:31.623571 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.623546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:31.623906 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.623888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dtdw6\""
Sep 16 18:44:31.624156 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.624137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:31.626371 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.626351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.628916 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.628888    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.629030 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.628980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.629111 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.629053    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:31.629176 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.629155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bklf6\""
Sep 16 18:44:31.629436 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.629418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.648425 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.648405    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:31.651319 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.651260    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-crvv9"
Sep 16 18:44:31.661068 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.661042    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-crvv9"
Sep 16 18:44:31.706066 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.706035    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:31.720514 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-dbus\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.720639 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysconfig\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.720639 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-sys\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.720639 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cni-binary-copy\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.720639 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7sc\" (UniqueName: \"kubernetes.io/projected/d7a2e49a-5573-433a-aab1-84a96e43720a-kube-api-access-8r7sc\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-netns\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-ovn\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-bin\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f7fa222-cc1f-402d-bd83-cc799812dfc0-host-slash\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-run\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.720838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-kubelet\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-cnibin\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-hostroot\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-registration-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.720980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-modprobe-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.721149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-netns\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.721412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f7fa222-cc1f-402d-bd83-cc799812dfc0-host-slash\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.721412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.721412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-netns\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-registration-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-bin\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-ovn\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-etc-kubernetes\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-kubelet\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-conf\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-agent-certs\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.721604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-var-lib-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-log-socket\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6cb\" (UniqueName: \"kubernetes.io/projected/39723fd9-72ef-45a2-8c31-54d4384ee595-kube-api-access-xp6cb\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtbb8\" (UniqueName: \"kubernetes.io/projected/82d7d29b-e205-4f7e-9285-f2cb1228acc6-kube-api-access-qtbb8\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-system-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-config\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.721870 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-systemd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-env-overrides\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkzl\" (UniqueName: \"kubernetes.io/projected/317d9c48-0c40-4699-ab85-78fe0f29a92a-kube-api-access-lvkzl\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-system-cni-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-kubelet-config\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.721995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-socket-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9mb\" (UniqueName: \"kubernetes.io/projected/e1506a9e-c134-411b-8a43-0ff0bd857d2c-kube-api-access-zf9mb\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-tmp\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-bin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-node-log\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5cb\" (UniqueName: \"kubernetes.io/projected/6b0b751f-bb0a-42fc-927a-754529fd0400-kube-api-access-sj5cb\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.722189 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-socket-dir-parent\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-k8s-cni-cncf-io\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-var-lib-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-multus\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-node-log\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-kubelet\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rp5\" (UniqueName: \"kubernetes.io/projected/06ca552f-d354-4b4f-9f3a-a30781b8bd71-kube-api-access-d6rp5\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-systemd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sln\" (UniqueName: \"kubernetes.io/projected/5f7fa222-cc1f-402d-bd83-cc799812dfc0-kube-api-access-n4sln\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-log-socket\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-lib-modules\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-conf-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-device-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-etc-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317d9c48-0c40-4699-ab85-78fe0f29a92a-tmp-dir\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-kubernetes\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.722686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-systemd\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-var-lib-kubelet\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1506a9e-c134-411b-8a43-0ff0bd857d2c-host\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-etc-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.722749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-socket-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-env-overrides\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-tuned\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cnibin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-device-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-daemon-config\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1506a9e-c134-411b-8a43-0ff0bd857d2c-host\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-multus-certs\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-systemd-units\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-sys-fs\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-slash\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.723428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-netd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-script-lib\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-host\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317d9c48-0c40-4699-ab85-78fe0f29a92a-hosts-file\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp4s\" (UniqueName: \"kubernetes.io/projected/d79c774c-df1c-4c16-bb55-a4a481ab692b-kube-api-access-5gp4s\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39723fd9-72ef-45a2-8c31-54d4384ee595-ovn-node-metrics-cert\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1506a9e-c134-411b-8a43-0ff0bd857d2c-serviceca\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-config\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f7fa222-cc1f-402d-bd83-cc799812dfc0-iptables-alerter-script\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-konnectivity-ca\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-run-openvswitch\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.723761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-konnectivity-ca\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f7fa222-cc1f-402d-bd83-cc799812dfc0-iptables-alerter-script\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-systemd-units\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-cni-netd\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39723fd9-72ef-45a2-8c31-54d4384ee595-host-slash\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d7a2e49a-5573-433a-aab1-84a96e43720a-sys-fs\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39723fd9-72ef-45a2-8c31-54d4384ee595-ovnkube-script-lib\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.724770 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1506a9e-c134-411b-8a43-0ff0bd857d2c-serviceca\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.725208 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.724953    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:31.729078 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.728914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39723fd9-72ef-45a2-8c31-54d4384ee595-ovn-node-metrics-cert\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.729231 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.729213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed3d6ac7-7893-4b5d-b0f3-ad05260e4556-agent-certs\") pod \"konnectivity-agent-ftd65\" (UID: \"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556\") " pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.734929 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.734910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9mb\" (UniqueName: \"kubernetes.io/projected/e1506a9e-c134-411b-8a43-0ff0bd857d2c-kube-api-access-zf9mb\") pod \"node-ca-zfpxr\" (UID: \"e1506a9e-c134-411b-8a43-0ff0bd857d2c\") " pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.735407 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.735376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7sc\" (UniqueName: \"kubernetes.io/projected/d7a2e49a-5573-433a-aab1-84a96e43720a-kube-api-access-8r7sc\") pod \"aws-ebs-csi-driver-node-zzc2w\" (UID: \"d7a2e49a-5573-433a-aab1-84a96e43720a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.735490 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.735406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sln\" (UniqueName: \"kubernetes.io/projected/5f7fa222-cc1f-402d-bd83-cc799812dfc0-kube-api-access-n4sln\") pod \"iptables-alerter-m977x\" (UID: \"5f7fa222-cc1f-402d-bd83-cc799812dfc0\") " pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.735490 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.735413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6cb\" (UniqueName: \"kubernetes.io/projected/39723fd9-72ef-45a2-8c31-54d4384ee595-kube-api-access-xp6cb\") pod \"ovnkube-node-pqmkz\" (UID: \"39723fd9-72ef-45a2-8c31-54d4384ee595\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.744237 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.744196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"cd7072b9c9593b3032f24b69cb3ef8c2","Type":"ContainerStarted","Data":"a4cab636f01bd4737bfed14edbdf7099c32bb18def94dfbb475bafea8b2e3c84"}
Sep 16 18:44:31.745503 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.745477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"6a2fa8135ce3bbe90bb18a946454dc57","Type":"ContainerStarted","Data":"7c5314e8d9ba5545cc2ef4c8e263010c697498db8c268a5d4bc08687a69c5597"}
Sep 16 18:44:31.824767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-dbus\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysconfig\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-sys\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cni-binary-copy\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-run\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-cnibin\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.824923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-sys\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-hostroot\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysconfig\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-dbus\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-cnibin\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-modprobe-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-netns\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-etc-kubernetes\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-modprobe-d\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-netns\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.824991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-run\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-conf\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-sysctl-conf\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.825309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-etc-kubernetes\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtbb8\" (UniqueName: \"kubernetes.io/projected/82d7d29b-e205-4f7e-9285-f2cb1228acc6-kube-api-access-qtbb8\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-system-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.825394    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkzl\" (UniqueName: \"kubernetes.io/projected/317d9c48-0c40-4699-ab85-78fe0f29a92a-kube-api-access-lvkzl\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-system-cni-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.825457    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.325435024 +0000 UTC m=+3.070930815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-system-cni-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-hostroot\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-kubelet-config\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cni-binary-copy\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-tmp\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-bin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5cb\" (UniqueName: \"kubernetes.io/projected/6b0b751f-bb0a-42fc-927a-754529fd0400-kube-api-access-sj5cb\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.826264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-socket-dir-parent\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-k8s-cni-cncf-io\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-k8s-cni-cncf-io\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9f788b9-e314-4ef7-ab91-cf007fc45061-kubelet-config\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.825761    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-system-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.825799    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.325787513 +0000 UTC m=+3.071283290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-cni-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-bin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.825972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-socket-dir-parent\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-multus\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-kubelet\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rp5\" (UniqueName: \"kubernetes.io/projected/06ca552f-d354-4b4f-9f3a-a30781b8bd71-kube-api-access-d6rp5\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-kubelet\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-var-lib-cni-multus\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-lib-modules\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827000 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-conf-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317d9c48-0c40-4699-ab85-78fe0f29a92a-tmp-dir\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-lib-modules\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-conf-dir\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-kubernetes\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-systemd\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-var-lib-kubelet\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-kubernetes\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-tuned\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-systemd\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-var-lib-kubelet\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cnibin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-daemon-config\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-multus-certs\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-cnibin\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.827715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-host\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06ca552f-d354-4b4f-9f3a-a30781b8bd71-host-run-multus-certs\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317d9c48-0c40-4699-ab85-78fe0f29a92a-tmp-dir\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b0b751f-bb0a-42fc-927a-754529fd0400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317d9c48-0c40-4699-ab85-78fe0f29a92a-hosts-file\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp4s\" (UniqueName: \"kubernetes.io/projected/d79c774c-df1c-4c16-bb55-a4a481ab692b-kube-api-access-5gp4s\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317d9c48-0c40-4699-ab85-78fe0f29a92a-hosts-file\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d7d29b-e205-4f7e-9285-f2cb1228acc6-host\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b0b751f-bb0a-42fc-927a-754529fd0400-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.828333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.826923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06ca552f-d354-4b4f-9f3a-a30781b8bd71-multus-daemon-config\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.828787 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.828768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-etc-tuned\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.828842 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.828820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d7d29b-e205-4f7e-9285-f2cb1228acc6-tmp\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.833478 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.833456    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:31.833582 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.833484    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:31.833582 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.833495    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.833582 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:31.833555    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.333542011 +0000 UTC m=+3.079037813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.835918 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.835858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtbb8\" (UniqueName: \"kubernetes.io/projected/82d7d29b-e205-4f7e-9285-f2cb1228acc6-kube-api-access-qtbb8\") pod \"tuned-fxkql\" (UID: \"82d7d29b-e205-4f7e-9285-f2cb1228acc6\") " pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:31.838855 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.838830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkzl\" (UniqueName: \"kubernetes.io/projected/317d9c48-0c40-4699-ab85-78fe0f29a92a-kube-api-access-lvkzl\") pod \"node-resolver-bf6tv\" (UID: \"317d9c48-0c40-4699-ab85-78fe0f29a92a\") " pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.841320 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.841293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp4s\" (UniqueName: \"kubernetes.io/projected/d79c774c-df1c-4c16-bb55-a4a481ab692b-kube-api-access-5gp4s\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:31.842754 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.842729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rp5\" (UniqueName: \"kubernetes.io/projected/06ca552f-d354-4b4f-9f3a-a30781b8bd71-kube-api-access-d6rp5\") pod \"multus-ghj9s\" (UID: \"06ca552f-d354-4b4f-9f3a-a30781b8bd71\") " pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.842754 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.842750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5cb\" (UniqueName: \"kubernetes.io/projected/6b0b751f-bb0a-42fc-927a-754529fd0400-kube-api-access-sj5cb\") pod \"multus-additional-cni-plugins-cjsrm\" (UID: \"6b0b751f-bb0a-42fc-927a-754529fd0400\") " pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.911088 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.911059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:31.921130 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.921106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w"
Sep 16 18:44:31.932120 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.932101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:31.940897 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.940877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m977x"
Sep 16 18:44:31.949815 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.949783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zfpxr"
Sep 16 18:44:31.959914 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.959892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghj9s"
Sep 16 18:44:31.967922 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.967895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bf6tv"
Sep 16 18:44:31.973905 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.973875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cjsrm"
Sep 16 18:44:31.980952 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:31.980930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fxkql"
Sep 16 18:44:32.330951 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:32.330879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:32.330951 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:32.330943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:32.331163 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.331040    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:32.331163 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.331121    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.331106108 +0000 UTC m=+4.076601885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:32.331163 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.331041    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:32.331294 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.331211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.331197448 +0000 UTC m=+4.076693226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:32.432344 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:32.432308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:32.432528 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.432506    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:32.432628 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.432540    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:32.432628 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.432558    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:32.432723 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:32.432633    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.432611024 +0000 UTC m=+4.178106819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:32.664718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:32.664625    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:32 +0000 UTC" deadline="2028-06-27 08:09:16.346797226 +0000 UTC"
Sep 16 18:44:32.664718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:32.664663    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15589h24m43.682139858s"
Sep 16 18:44:32.772712 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.772574    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317d9c48_0c40_4699_ab85_78fe0f29a92a.slice/crio-e71bfc714e7b7a271b8b5192cca4aea83c5a1ea5c96ed80577e7e72213b4a11c WatchSource:0}: Error finding container e71bfc714e7b7a271b8b5192cca4aea83c5a1ea5c96ed80577e7e72213b4a11c: Status 404 returned error can't find the container with id e71bfc714e7b7a271b8b5192cca4aea83c5a1ea5c96ed80577e7e72213b4a11c
Sep 16 18:44:32.775527 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.775425    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0b751f_bb0a_42fc_927a_754529fd0400.slice/crio-3b19ee982d184b1b1fa711ae5ff5bca34a4bc08c85157ef4bfc10ab62911a2d4 WatchSource:0}: Error finding container 3b19ee982d184b1b1fa711ae5ff5bca34a4bc08c85157ef4bfc10ab62911a2d4: Status 404 returned error can't find the container with id 3b19ee982d184b1b1fa711ae5ff5bca34a4bc08c85157ef4bfc10ab62911a2d4
Sep 16 18:44:32.775913 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.775811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d7d29b_e205_4f7e_9285_f2cb1228acc6.slice/crio-303daa3cf9450cf9d16ae549898a59c6a1d21aa2cb76b5833bdda08d5a6ca4a6 WatchSource:0}: Error finding container 303daa3cf9450cf9d16ae549898a59c6a1d21aa2cb76b5833bdda08d5a6ca4a6: Status 404 returned error can't find the container with id 303daa3cf9450cf9d16ae549898a59c6a1d21aa2cb76b5833bdda08d5a6ca4a6
Sep 16 18:44:32.779480 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.779460    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1506a9e_c134_411b_8a43_0ff0bd857d2c.slice/crio-2af4169f2fe88ffb4ed851c7ab0ad59fbd160766fef0f183345f978b8fd43528 WatchSource:0}: Error finding container 2af4169f2fe88ffb4ed851c7ab0ad59fbd160766fef0f183345f978b8fd43528: Status 404 returned error can't find the container with id 2af4169f2fe88ffb4ed851c7ab0ad59fbd160766fef0f183345f978b8fd43528
Sep 16 18:44:32.780858 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.780611    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39723fd9_72ef_45a2_8c31_54d4384ee595.slice/crio-12d464e26b07ba0111bf4a8b4b265d7c872bbb5e62f83d64dc6367c96429db08 WatchSource:0}: Error finding container 12d464e26b07ba0111bf4a8b4b265d7c872bbb5e62f83d64dc6367c96429db08: Status 404 returned error can't find the container with id 12d464e26b07ba0111bf4a8b4b265d7c872bbb5e62f83d64dc6367c96429db08
Sep 16 18:44:32.782462 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:44:32.782303    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7fa222_cc1f_402d_bd83_cc799812dfc0.slice/crio-d72e9ecfb1f0e17a5bb6dbcefbf1146f4d8b724401e4a206a212f0d4df53cbde WatchSource:0}: Error finding container d72e9ecfb1f0e17a5bb6dbcefbf1146f4d8b724401e4a206a212f0d4df53cbde: Status 404 returned error can't find the container with id d72e9ecfb1f0e17a5bb6dbcefbf1146f4d8b724401e4a206a212f0d4df53cbde
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.339678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.339794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.339931    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.340000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.339978838 +0000 UTC m=+6.085474635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.340440    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:33.340530 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.340495    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.340478468 +0000 UTC m=+6.085974249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:33.441320 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.440732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:33.441320 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.440887    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:33.441320 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.440909    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:33.441320 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.440921    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:33.441320 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.440977    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.440959928 +0000 UTC m=+6.186455705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:33.665556 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.665436    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:32 +0000 UTC" deadline="2028-03-31 04:14:48.604011227 +0000 UTC"
Sep 16 18:44:33.665556 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.665500    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13473h30m14.938515261s"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.742115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.742252    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.742651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.742760    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.742826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:33.743172 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:33.742897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:33.771170 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.770910    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd7072b9c9593b3032f24b69cb3ef8c2" containerID="31c9115bc8ad46584eb525a3ab9cc10c36685ef6a06c411f446a52ddb75433a4" exitCode=0
Sep 16 18:44:33.771170 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.771013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"cd7072b9c9593b3032f24b69cb3ef8c2","Type":"ContainerDied","Data":"31c9115bc8ad46584eb525a3ab9cc10c36685ef6a06c411f446a52ddb75433a4"}
Sep 16 18:44:33.780230 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.779814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghj9s" event={"ID":"06ca552f-d354-4b4f-9f3a-a30781b8bd71","Type":"ContainerStarted","Data":"c5a7ad633a11148275a4af4ecae1257a124a676dea0f633ae0b7ab9259217f37"}
Sep 16 18:44:33.788773 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.788747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"12d464e26b07ba0111bf4a8b4b265d7c872bbb5e62f83d64dc6367c96429db08"}
Sep 16 18:44:33.801079 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.801040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zfpxr" event={"ID":"e1506a9e-c134-411b-8a43-0ff0bd857d2c","Type":"ContainerStarted","Data":"2af4169f2fe88ffb4ed851c7ab0ad59fbd160766fef0f183345f978b8fd43528"}
Sep 16 18:44:33.808543 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.808519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxkql" event={"ID":"82d7d29b-e205-4f7e-9285-f2cb1228acc6","Type":"ContainerStarted","Data":"303daa3cf9450cf9d16ae549898a59c6a1d21aa2cb76b5833bdda08d5a6ca4a6"}
Sep 16 18:44:33.814400 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.814373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerStarted","Data":"3b19ee982d184b1b1fa711ae5ff5bca34a4bc08c85157ef4bfc10ab62911a2d4"}
Sep 16 18:44:33.821892 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.821866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bf6tv" event={"ID":"317d9c48-0c40-4699-ab85-78fe0f29a92a","Type":"ContainerStarted","Data":"e71bfc714e7b7a271b8b5192cca4aea83c5a1ea5c96ed80577e7e72213b4a11c"}
Sep 16 18:44:33.840224 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.837265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" event={"ID":"6a2fa8135ce3bbe90bb18a946454dc57","Type":"ContainerStarted","Data":"18d9aceaca9ff50cfcfb87607467e41245b3f99a478cc7937c268aed3dc71067"}
Sep 16 18:44:33.845701 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.845620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w" event={"ID":"d7a2e49a-5573-433a-aab1-84a96e43720a","Type":"ContainerStarted","Data":"17159dae736640e168b061e23b54f48560de04c569077f9dab990a97adc20eac"}
Sep 16 18:44:33.856478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.856452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftd65" event={"ID":"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556","Type":"ContainerStarted","Data":"ad5a2414b48413f0c01d3e699545377298cfd7174a4223b99a0ba473f3dc53b0"}
Sep 16 18:44:33.861191 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:33.861165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m977x" event={"ID":"5f7fa222-cc1f-402d-bd83-cc799812dfc0","Type":"ContainerStarted","Data":"d72e9ecfb1f0e17a5bb6dbcefbf1146f4d8b724401e4a206a212f0d4df53cbde"}
Sep 16 18:44:34.881607 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:34.881569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" event={"ID":"cd7072b9c9593b3032f24b69cb3ef8c2","Type":"ContainerStarted","Data":"a2199db3e3f5849bbd28f5e13ae8e2bee191cc1ccfcbc49d4aa671597d2e83f6"}
Sep 16 18:44:34.902363 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:34.902183    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-253.ec2.internal" podStartSLOduration=2.902163374 podStartE2EDuration="2.902163374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:33.854497186 +0000 UTC m=+4.599993001" watchObservedRunningTime="2026-09-16 18:44:34.902163374 +0000 UTC m=+5.647659170"
Sep 16 18:44:35.358838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.358748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:35.358838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.358812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:35.359120 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.359011    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:35.359120 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.359096    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.359076133 +0000 UTC m=+10.104571916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:35.359519 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.359497    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:35.359595 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.359557    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.359540147 +0000 UTC m=+10.105035942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:35.459400 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.459364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:35.459595 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.459580    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:35.459662 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.459601    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:35.459662 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.459615    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:35.459778 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.459674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.459653627 +0000 UTC m=+10.205149426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:35.742343 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.742258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:35.742520 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.742412    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:35.742856 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.742835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:35.742951 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.742934    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:35.743017 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:35.742995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:35.743102 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:35.743083    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:37.739221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:37.739346    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:37.740098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:37.740196    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:37.740249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:37.741881 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:37.740329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:39.390216 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.390175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:39.390713 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.390235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:39.390713 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.390330    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.390713 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.390364    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.390713 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.390414    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.390391263 +0000 UTC m=+18.135887060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.390713 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.390434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.390424772 +0000 UTC m=+18.135920561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.490637 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.490593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:39.490830 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.490803    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:39.490830 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.490832    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:39.491051 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.490846    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:39.491051 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.490906    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.490883684 +0000 UTC m=+18.236379464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:39.739963 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.739722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:39.741279 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.740907    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:39.741279 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.740758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:39.741279 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.741040    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:39.741279 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:39.740714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:39.741279 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:39.741207    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:41.741333 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:41.741298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:41.741784 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:41.741414    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:41.741852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:41.741797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:41.741902 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:41.741887    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:41.741958 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:41.741941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:41.742005 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:41.741991    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:43.739241 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:43.739199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:43.739686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:43.739215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:43.739686 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:43.739320    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:43.739686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:43.739343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:43.739686 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:43.739459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:43.739686 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:43.739548    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:45.739772 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:45.739736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:45.740227 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:45.739785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:45.740227 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:45.739745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:45.740227 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:45.739870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:45.740227 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:45.739971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:45.740227 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:45.740064    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:47.449499 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.449468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:47.450006 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.449533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:47.450006 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.449603    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:47.450006 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.449635    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:47.450006 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.449665    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.449649561 +0000 UTC m=+34.195145341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:47.450006 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.449691    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.449674199 +0000 UTC m=+34.195169978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:47.550472 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.550442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:47.550617 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.550550    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:47.550617 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.550562    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:47.550617 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.550570    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:47.550617 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.550614    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.550601468 +0000 UTC m=+34.296097246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.739505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.739539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:47.739647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.739642    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.739717    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:47.740048 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:47.739793    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:49.741812 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:49.741777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:49.742233 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:49.741835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:49.742233 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:49.741909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:49.742233 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:49.741908    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:49.742233 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:49.742059    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:49.742233 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:49.742145    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:51.739886 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.739726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:51.740655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.739723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:51.740655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.739784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:51.740655 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:51.739967    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:51.740655 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:51.740062    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:51.740655 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:51.740152    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:51.923725 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.923697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"c6ed8e5cc944fa32ef1786933438d87ef9f1909ac2c469c857182008574ce903"}
Sep 16 18:44:51.923825 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.923764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"52b1c5fe4cd2571110b252e0f3062cc75d21b600d726b2b65b0a4fa0a518e1d8"}
Sep 16 18:44:51.923825 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.923801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"52047d1a22a56c5f419990c8574297797d212acce55d03ce4636238c7b1bd781"}
Sep 16 18:44:51.923921 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.923830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"a349b633f4bf2debf606ed17a83586f43f0f0fd4037503a0a82dd872b0d6d8fb"}
Sep 16 18:44:51.925002 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.924963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zfpxr" event={"ID":"e1506a9e-c134-411b-8a43-0ff0bd857d2c","Type":"ContainerStarted","Data":"14a7f04ca89a489f1350b85a4ba5d46d3b5d64ba1e9cf24f5eb95ae7a351b77f"}
Sep 16 18:44:51.926336 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.926316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fxkql" event={"ID":"82d7d29b-e205-4f7e-9285-f2cb1228acc6","Type":"ContainerStarted","Data":"101676afc350d4dffe16cbdc4b1ecd8be1576e146b448167469e237ba1e759a0"}
Sep 16 18:44:51.927858 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.927839    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="e71f5be200eac5d84b64927b406828338ba740098044c8bb6cfa9a640a18e114" exitCode=0
Sep 16 18:44:51.927943 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.927901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"e71f5be200eac5d84b64927b406828338ba740098044c8bb6cfa9a640a18e114"}
Sep 16 18:44:51.929482 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.929460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bf6tv" event={"ID":"317d9c48-0c40-4699-ab85-78fe0f29a92a","Type":"ContainerStarted","Data":"7439614b4635ecfbb75fc2e26121fe0deefbc2764fd22ca061e89f983f699e67"}
Sep 16 18:44:51.930979 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.930956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w" event={"ID":"d7a2e49a-5573-433a-aab1-84a96e43720a","Type":"ContainerStarted","Data":"a642c7633b029b7b68791b305a78fab3fa7f0660e8fdf5712c98c2cc30df1a5d"}
Sep 16 18:44:51.932346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.932292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftd65" event={"ID":"ed3d6ac7-7893-4b5d-b0f3-ad05260e4556","Type":"ContainerStarted","Data":"d090172337b465f7af383547b97cf818d204ae1c7ccf00c8a67ecc38fbfa3885"}
Sep 16 18:44:51.933760 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.933741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghj9s" event={"ID":"06ca552f-d354-4b4f-9f3a-a30781b8bd71","Type":"ContainerStarted","Data":"f48c329f1cc25cfb9d728d00afc5b6a27c743f74f68c8ea0a7154f85970a1965"}
Sep 16 18:44:51.943154 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.941025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zfpxr" podStartSLOduration=3.326874108 podStartE2EDuration="21.941009814s" totalImagesPullingTime="18.614135706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.940920207 +0000 UTC m=+22.686416005" watchObservedRunningTime="2026-09-16 18:44:51.941009814 +0000 UTC m=+22.686505625"
Sep 16 18:44:51.943154 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.941147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-253.ec2.internal" podStartSLOduration=20.941140505 podStartE2EDuration="20.941140505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:34.90323868 +0000 UTC m=+5.648734506" watchObservedRunningTime="2026-09-16 18:44:51.941140505 +0000 UTC m=+22.686636303"
Sep 16 18:44:51.982665 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.982580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bf6tv" podStartSLOduration=2.361677131 podStartE2EDuration="20.982568357s" totalImagesPullingTime="18.620891226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.982542114 +0000 UTC m=+22.728037911" watchObservedRunningTime="2026-09-16 18:44:51.982568357 +0000 UTC m=+22.728064154"
Sep 16 18:44:51.982734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:51.982692    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ghj9s" podStartSLOduration=3.322648319 podStartE2EDuration="21.982685643s" totalImagesPullingTime="18.660037324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.961912668 +0000 UTC m=+22.707408475" watchObservedRunningTime="2026-09-16 18:44:51.982685643 +0000 UTC m=+22.728181439"
Sep 16 18:44:52.042477 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.042434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fxkql" podStartSLOduration=2.42295379 podStartE2EDuration="21.042405835s" totalImagesPullingTime="18.619452045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.022002491 +0000 UTC m=+22.767498279" watchObservedRunningTime="2026-09-16 18:44:52.042405835 +0000 UTC m=+22.787901662"
Sep 16 18:44:52.042695 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.042677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ftd65" podStartSLOduration=10.703015921 podStartE2EDuration="22.042672404s" totalImagesPullingTime="11.339656483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.042548573 +0000 UTC m=+22.788044370" watchObservedRunningTime="2026-09-16 18:44:52.042672404 +0000 UTC m=+22.788168200"
Sep 16 18:44:52.125689 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.125658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:52.196059 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.195959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:52.941106 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.940890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"cbec29e381422b00021869ee33036a924eaf768b552fd77b35044b489628035f"}
Sep 16 18:44:52.941534 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.941207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"8a7b58befb286ec01610cfbdd94158df0d6522e99890187b01e2926acb874b14"}
Sep 16 18:44:52.942541 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.942483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m977x" event={"ID":"5f7fa222-cc1f-402d-bd83-cc799812dfc0","Type":"ContainerStarted","Data":"a4c33b8b6d28cd85e01ec12128fe371c9f6ed6486ce3acdc7df582ec71494bc0"}
Sep 16 18:44:52.957548 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:52.957460    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m977x" podStartSLOduration=4.34907166 podStartE2EDuration="22.957442415s" totalImagesPullingTime="18.608370755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.956908119 +0000 UTC m=+23.702403916" watchObservedRunningTime="2026-09-16 18:44:52.957442415 +0000 UTC m=+23.702938214"
Sep 16 18:44:53.007374 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.007351    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:53.086838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.086808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:53.181063 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.181033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:53.683910 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.683788    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:53.007370037Z","UUID":"85c369f6-873f-4a08-943f-9764e8867b29","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:53.685678 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.685639    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:53.685678 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.685676    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:53.739230 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.739200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:53.739391 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:53.739302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:53.739666 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.739650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:53.739736 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:53.739722    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:53.739857 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.739844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:53.739931 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:53.739914    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:53.910909 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.910876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:53.911505 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.911473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:53.947836 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.947152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w" event={"ID":"d7a2e49a-5573-433a-aab1-84a96e43720a","Type":"ContainerStarted","Data":"b1256a181bc86bddbec8a892ac977233dde4dcb3236fd129aff119266f58ca0a"}
Sep 16 18:44:53.947836 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.947743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:53.948611 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:53.948591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ftd65"
Sep 16 18:44:54.091489 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:54.091456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:54.954262 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:54.954221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"f2fee1ce5d97e3401c8637221be6f05e9751b08bf604b72544596a7f591410e2"}
Sep 16 18:44:54.956641 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:54.956610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w" event={"ID":"d7a2e49a-5573-433a-aab1-84a96e43720a","Type":"ContainerStarted","Data":"fdd70681bf9a97696d0b1613d7cbd142e13c3f55f5409a9fa529b5ea0c8d7a4a"}
Sep 16 18:44:54.976838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:54.976796    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzc2w" podStartSLOduration=4.058000546 podStartE2EDuration="24.976783201s" totalImagesPullingTime="20.918782655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:54.976471545 +0000 UTC m=+25.721967345" watchObservedRunningTime="2026-09-16 18:44:54.976783201 +0000 UTC m=+25.722278998"
Sep 16 18:44:55.739897 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:55.739868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:55.740107 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:55.739880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:55.740107 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:55.740016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:55.740107 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:55.740017    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:55.740268 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:55.740103    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:55.740268 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:55.740191    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:55.820521 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:55.820453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:55.907628 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:55.907596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.109052 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.108963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.208616 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.208581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.965627 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.965107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" event={"ID":"39723fd9-72ef-45a2-8c31-54d4384ee595","Type":"ContainerStarted","Data":"538a3d72c9e77341ebacd955d5598e09dcb62c68b7daf33f9e38eed965d535a0"}
Sep 16 18:44:56.965627 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.965465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:56.965627 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.965526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:56.965627 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.965572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:56.967366 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.967332    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="8e32b68e0579765e252189c3431ed7f8077525c1c8902dd64d15a9fd80bf04c7" exitCode=0
Sep 16 18:44:56.967475 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.967368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"8e32b68e0579765e252189c3431ed7f8077525c1c8902dd64d15a9fd80bf04c7"}
Sep 16 18:44:56.981131 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.981111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:56.981246 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.981231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:44:56.998977 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:56.998939    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz" podStartSLOduration=8.329928165 podStartE2EDuration="26.998927187s" totalImagesPullingTime="18.668999022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.99722917 +0000 UTC m=+27.742724967" watchObservedRunningTime="2026-09-16 18:44:56.998927187 +0000 UTC m=+27.744422984"
Sep 16 18:44:57.739111 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:57.739071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:57.739848 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:57.739185    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:57.739848 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:57.739199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:57.739848 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:57.739279    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:57.739848 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:57.739305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:57.739848 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:57.739385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:58.245785 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.245603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nq5t"]
Sep 16 18:44:58.245888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.245876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:58.245962 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.245945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h44vj"]
Sep 16 18:44:58.246034 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:58.246012    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:58.246034 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.246031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:58.246175 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:58.246105    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:58.247633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.247609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svtlg"]
Sep 16 18:44:58.248046 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.248028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:58.249127 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:58.249096    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:58.442531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.442504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:58.518912 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.518889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:58.973495 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.973464    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="93ce0f3ea9c2ecee34bad7701cd8d66c52a46f53b2a934e5f88dcd0559467f5a" exitCode=0
Sep 16 18:44:58.974022 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:58.973545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"93ce0f3ea9c2ecee34bad7701cd8d66c52a46f53b2a934e5f88dcd0559467f5a"}
Sep 16 18:44:59.740915 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:59.740882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:44:59.741091 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:59.741007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:44:59.741091 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:59.741034    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:44:59.741184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:59.741100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:44:59.741184 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:59.741101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:44:59.741298 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:44:59.741278    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:44:59.804754 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:44:59.804727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.981197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:00.981159    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="f6dee86b24f7c1c3a6be99878747be15959a3ef7c84ea491ee4b81f10261f4c8" exitCode=0
Sep 16 18:45:00.981197 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:00.981204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"f6dee86b24f7c1c3a6be99878747be15959a3ef7c84ea491ee4b81f10261f4c8"}
Sep 16 18:45:01.738959 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:01.738919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:01.739194 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:01.738919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:01.739194 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:01.739090    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:45:01.739308 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:01.739195    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:45:01.739308 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:01.739270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:01.739424 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:01.739326    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:45:03.462265 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.462231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:03.462745 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.462276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:03.462745 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.462399    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.462745 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.462405    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.462745 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.462452    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret podName:e9f788b9-e314-4ef7-ab91-cf007fc45061 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.462439436 +0000 UTC m=+66.207935212 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret") pod "global-pull-secret-syncer-6nq5t" (UID: "e9f788b9-e314-4ef7-ab91-cf007fc45061") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.462745 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.462477    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs podName:d79c774c-df1c-4c16-bb55-a4a481ab692b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.46245975 +0000 UTC m=+66.207955526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs") pod "network-metrics-daemon-svtlg" (UID: "d79c774c-df1c-4c16-bb55-a4a481ab692b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.563093 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.563050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:03.563277 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.563246    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:03.563277 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.563272    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:03.563374 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.563284    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf4k2 for pod openshift-network-diagnostics/network-check-target-h44vj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.563421 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.563411    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2 podName:a62f8bbe-eca2-454e-9254-cae6e118f310 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.563390271 +0000 UTC m=+66.308886047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf4k2" (UniqueName: "kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2") pod "network-check-target-h44vj" (UID: "a62f8bbe-eca2-454e-9254-cae6e118f310") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.739895 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.739804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:03.740063 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.739804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:03.740063 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.739944    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svtlg" podUID="d79c774c-df1c-4c16-bb55-a4a481ab692b"
Sep 16 18:45:03.740063 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:03.739813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:03.740063 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.740011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h44vj" podUID="a62f8bbe-eca2-454e-9254-cae6e118f310"
Sep 16 18:45:03.740276 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:03.740102    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nq5t" podUID="e9f788b9-e314-4ef7-ab91-cf007fc45061"
Sep 16 18:45:04.018669 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.018584    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-253.ec2.internal" event="NodeReady"
Sep 16 18:45:04.018828 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.018765    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:04.076996 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.076957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2dvs4"]
Sep 16 18:45:04.079530 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.079506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.081226 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.081207    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n8jt7"]
Sep 16 18:45:04.082294 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.082271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:04.082643 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.082623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wx8jg\""
Sep 16 18:45:04.082728 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.082708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:04.082794 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.082763    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:04.083131 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.083111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.086489 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.086466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:04.086705 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.086689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:04.087088 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.086939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7kfw\""
Sep 16 18:45:04.093612 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.093591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2dvs4"]
Sep 16 18:45:04.099976 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.099914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8jt7"]
Sep 16 18:45:04.166878 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.166844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53030e7e-f330-4608-9405-184801d5a706-cert\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.166878 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.166877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-tmp-dir\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.167142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.166907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxn7f\" (UniqueName: \"kubernetes.io/projected/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-kube-api-access-hxn7f\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.167142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.166964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-config-volume\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.167142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.167009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.167142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.167048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdrv\" (UniqueName: \"kubernetes.io/projected/53030e7e-f330-4608-9405-184801d5a706-kube-api-access-ssdrv\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.244096 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.244033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:04.267970 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.267932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-config-volume\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.267970 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.267972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.268215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdrv\" (UniqueName: \"kubernetes.io/projected/53030e7e-f330-4608-9405-184801d5a706-kube-api-access-ssdrv\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.268215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53030e7e-f330-4608-9405-184801d5a706-cert\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.268215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-tmp-dir\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.268215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxn7f\" (UniqueName: \"kubernetes.io/projected/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-kube-api-access-hxn7f\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.268215 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:04.268157    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:04.268451 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:04.268254    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.768226876 +0000 UTC m=+35.513722651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:04.268492 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-tmp-dir\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.268715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.268660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-config-volume\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.273044 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.273017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53030e7e-f330-4608-9405-184801d5a706-cert\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.282026 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.282004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxn7f\" (UniqueName: \"kubernetes.io/projected/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-kube-api-access-hxn7f\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.282205 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.282181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdrv\" (UniqueName: \"kubernetes.io/projected/53030e7e-f330-4608-9405-184801d5a706-kube-api-access-ssdrv\") pod \"ingress-canary-2dvs4\" (UID: \"53030e7e-f330-4608-9405-184801d5a706\") " pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.342751 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.342720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:04.392764 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.392726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2dvs4"
Sep 16 18:45:04.771753 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:04.771714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:04.772207 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:04.771862    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:04.772207 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:04.771931    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.771915125 +0000 UTC m=+36.517410900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:05.452871 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.452613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2dvs4"]
Sep 16 18:45:05.458104 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:05.458068    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53030e7e_f330_4608_9405_184801d5a706.slice/crio-949ed30fa62913d9eb54cbb16b0ac877b57b323f2e32629aa77657d90cc5c625 WatchSource:0}: Error finding container 949ed30fa62913d9eb54cbb16b0ac877b57b323f2e32629aa77657d90cc5c625: Status 404 returned error can't find the container with id 949ed30fa62913d9eb54cbb16b0ac877b57b323f2e32629aa77657d90cc5c625
Sep 16 18:45:05.739873 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.739787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:05.740055 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.739792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:05.740125 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.739792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:05.743609 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:05.743734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvlxk\""
Sep 16 18:45:05.743734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rk9xq\""
Sep 16 18:45:05.743734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:05.743889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:05.743942 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.743911    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:05.759248 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.759225    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"]
Sep 16 18:45:05.761346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.761326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"
Sep 16 18:45:05.764972 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.764956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:05.764972 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.764966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:05.765344 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.765331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2gvms\""
Sep 16 18:45:05.776128 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.776102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"]
Sep 16 18:45:05.779481 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.779458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:05.779615 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:05.779601    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:05.779673 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:05.779661    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.779643109 +0000 UTC m=+38.525138885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:05.880774 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.880732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxjk\" (UniqueName: \"kubernetes.io/projected/1b10af34-ebe5-4e40-9cbd-e2497f30c59c-kube-api-access-bhxjk\") pod \"volume-data-source-validator-86695b7cc7-nwjbp\" (UID: \"1b10af34-ebe5-4e40-9cbd-e2497f30c59c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"
Sep 16 18:45:05.962578 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.962539    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"]
Sep 16 18:45:05.964621 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.964598    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6469ffddb8-xx5ck"]
Sep 16 18:45:05.964769 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.964752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:05.966676 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.966654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:05.968245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.968223    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:45:05.968842 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.968809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:45:05.968924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.968844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-drg7t\""
Sep 16 18:45:05.971750 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.969011    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:45:05.971750 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.969464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:45:05.972619 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.972362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:45:05.972619 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.972399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8hgf6\""
Sep 16 18:45:05.972619 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.972373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:45:05.972619 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.972475    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:45:05.972809 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.972739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:45:05.973370 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.973351    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:45:05.973513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.973499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:45:05.981960 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.981670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxjk\" (UniqueName: \"kubernetes.io/projected/1b10af34-ebe5-4e40-9cbd-e2497f30c59c-kube-api-access-bhxjk\") pod \"volume-data-source-validator-86695b7cc7-nwjbp\" (UID: \"1b10af34-ebe5-4e40-9cbd-e2497f30c59c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"
Sep 16 18:45:05.984372 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.984351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6469ffddb8-xx5ck"]
Sep 16 18:45:05.984949 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.984930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"]
Sep 16 18:45:05.995575 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.995500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2dvs4" event={"ID":"53030e7e-f330-4608-9405-184801d5a706","Type":"ContainerStarted","Data":"949ed30fa62913d9eb54cbb16b0ac877b57b323f2e32629aa77657d90cc5c625"}
Sep 16 18:45:05.998810 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.998782    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="368cde3bb0eebc8880a2b17d42e262d87f46b786d7d455296cb58fb63ae0f7a1" exitCode=0
Sep 16 18:45:05.998930 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:05.998837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"368cde3bb0eebc8880a2b17d42e262d87f46b786d7d455296cb58fb63ae0f7a1"}
Sep 16 18:45:06.004221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.004200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxjk\" (UniqueName: \"kubernetes.io/projected/1b10af34-ebe5-4e40-9cbd-e2497f30c59c-kube-api-access-bhxjk\") pod \"volume-data-source-validator-86695b7cc7-nwjbp\" (UID: \"1b10af34-ebe5-4e40-9cbd-e2497f30c59c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"
Sep 16 18:45:06.068474 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.068443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"
Sep 16 18:45:06.082368 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab59f219-af81-45d3-a712-d04595c124d7-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.082478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7sfn\" (UniqueName: \"kubernetes.io/projected/ab59f219-af81-45d3-a712-d04595c124d7-kube-api-access-p7sfn\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.082478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-stats-auth\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.082478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.082633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb9l\" (UniqueName: \"kubernetes.io/projected/d23aa9cd-9131-483a-994e-483cbec4153f-kube-api-access-nmb9l\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.082633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-default-certificate\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.082633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.082633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.082575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.097521 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.097496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.183508 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab59f219-af81-45d3-a712-d04595c124d7-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7sfn\" (UniqueName: \"kubernetes.io/projected/ab59f219-af81-45d3-a712-d04595c124d7-kube-api-access-p7sfn\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-stats-auth\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb9l\" (UniqueName: \"kubernetes.io/projected/d23aa9cd-9131-483a-994e-483cbec4153f-kube-api-access-nmb9l\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-default-certificate\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.183668 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.183660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.183879 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.183745    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:06.183879 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.183789    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:06.683776434 +0000 UTC m=+37.429272210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:06.184780 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.184255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab59f219-af81-45d3-a712-d04595c124d7-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.184780 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.184676    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:06.684655955 +0000 UTC m=+37.430151732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:06.184780 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.184702    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:06.184780 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.184755    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:06.684738292 +0000 UTC m=+37.430234072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:06.187004 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.186967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-stats-auth\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.187423 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.187406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-default-certificate\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.192784 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.192723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7sfn\" (UniqueName: \"kubernetes.io/projected/ab59f219-af81-45d3-a712-d04595c124d7-kube-api-access-p7sfn\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.193844 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.193820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb9l\" (UniqueName: \"kubernetes.io/projected/d23aa9cd-9131-483a-994e-483cbec4153f-kube-api-access-nmb9l\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.194332 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.194312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.301359 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.301323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.354060 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.354027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp"]
Sep 16 18:45:06.357359 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:06.357324    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b10af34_ebe5_4e40_9cbd_e2497f30c59c.slice/crio-d9968009c6078f95f5351204c513c8b6d76579ed317a786539e5a468ee0323b6 WatchSource:0}: Error finding container d9968009c6078f95f5351204c513c8b6d76579ed317a786539e5a468ee0323b6: Status 404 returned error can't find the container with id d9968009c6078f95f5351204c513c8b6d76579ed317a786539e5a468ee0323b6
Sep 16 18:45:06.369780 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.369756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.687734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.687695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.687924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.687760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:06.687924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:06.687784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:06.687924 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.687880    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:06.687924 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.687896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.687877088 +0000 UTC m=+38.433372874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:06.687924 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.687913    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:06.688183 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.687943    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.687926325 +0000 UTC m=+38.433422104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:06.688183 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:06.687963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.687952608 +0000 UTC m=+38.433448389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:07.005823 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.005742    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b0b751f-bb0a-42fc-927a-754529fd0400" containerID="0ddeceafb53d9917366f74acc49c38fe64db81ac3e957308a4e7195d5632e01d" exitCode=0
Sep 16 18:45:07.005823 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.005777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerDied","Data":"0ddeceafb53d9917366f74acc49c38fe64db81ac3e957308a4e7195d5632e01d"}
Sep 16 18:45:07.007168 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.007145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp" event={"ID":"1b10af34-ebe5-4e40-9cbd-e2497f30c59c","Type":"ContainerStarted","Data":"d9968009c6078f95f5351204c513c8b6d76579ed317a786539e5a468ee0323b6"}
Sep 16 18:45:07.697888 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.697706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:07.697888 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.697863    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.69784551 +0000 UTC m=+40.443341284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:07.698079 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.697901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:07.698079 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.697999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:07.698079 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.698054    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:07.698079 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.698070    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:07.698257 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.698106    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.698094855 +0000 UTC m=+40.443590630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:07.698257 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.698122    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.698114389 +0000 UTC m=+40.443610163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:07.798261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:07.798241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:07.798427 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.798414    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:07.798480 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:07.798471    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:11.798453609 +0000 UTC m=+42.543949574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:08.014964 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.014925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" event={"ID":"6b0b751f-bb0a-42fc-927a-754529fd0400","Type":"ContainerStarted","Data":"e91edd605777405316fd31d2386b380ed0cf1f5b7674dc9efc64e30b3b92f244"}
Sep 16 18:45:08.016627 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.016599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2dvs4" event={"ID":"53030e7e-f330-4608-9405-184801d5a706","Type":"ContainerStarted","Data":"0bd338382a486387f936f5576327888ba03fddbc99af5b2a8cc68b9b8c92b0d4"}
Sep 16 18:45:08.055776 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.055701    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cjsrm" podStartSLOduration=7.897462578 podStartE2EDuration="37.055679077s" totalImagesPullingTime="29.158216499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.054274557 +0000 UTC m=+38.799770355" watchObservedRunningTime="2026-09-16 18:45:08.055679077 +0000 UTC m=+38.801174877"
Sep 16 18:45:08.078305 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.078204    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2dvs4" podStartSLOduration=1.741389359 podStartE2EDuration="4.078182736s" totalImagesPullingTime="2.336793377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.077733589 +0000 UTC m=+38.823229387" watchObservedRunningTime="2026-09-16 18:45:08.078182736 +0000 UTC m=+38.823678535"
Sep 16 18:45:08.185701 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.185666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:08.260290 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:08.260259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.021127 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.021034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp" event={"ID":"1b10af34-ebe5-4e40-9cbd-e2497f30c59c","Type":"ContainerStarted","Data":"6c6ddf2eabfd0ca002c13edc45a4ec9b9691375a6299d3886bc24c7ff0254818"}
Sep 16 18:45:09.055306 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.055260    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-nwjbp" podStartSLOduration=1.786943935 podStartE2EDuration="4.055245968s" totalImagesPullingTime="2.268302033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:09.05511911 +0000 UTC m=+39.800614901" watchObservedRunningTime="2026-09-16 18:45:09.055245968 +0000 UTC m=+39.800741786"
Sep 16 18:45:09.271718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.271648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.382785 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.382753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.714965 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.714936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:09.714965 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.714968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:09.715187 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.715044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:09.715187 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:09.715097    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.715079605 +0000 UTC m=+44.460575380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:09.715187 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:09.715135    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:09.715187 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:09.715150    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:09.715399 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:09.715195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.71518338 +0000 UTC m=+44.460679156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:09.715399 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:09.715207    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.715201317 +0000 UTC m=+44.460697092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:09.886844 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.886802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.981547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:09.981469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.830820 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:11.830787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:11.831181 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:11.830930    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:11.831181 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:11.831004    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:19.830974893 +0000 UTC m=+50.576470667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:12.677656 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.677625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:12.788927 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.788900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:12.794901 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.794881    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fshll"]
Sep 16 18:45:12.839799 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.839773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fshll"]
Sep 16 18:45:12.840171 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.839874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:12.842500 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.842477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:45:12.842636 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.842585    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:45:12.842636 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.842497    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:45:12.842636 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.842498    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:45:12.842851 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.842649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8svg\""
Sep 16 18:45:12.880161 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.880138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:12.940431 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.940387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvrk\" (UniqueName: \"kubernetes.io/projected/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-api-access-fcvrk\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:12.940512 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.940420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-data-volume\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:12.940512 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.940451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:12.940512 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.940477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-crio-socket\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:12.940606 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:12.940545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.040938 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.040910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-crio-socket\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041068 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.040980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041123 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.041078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvrk\" (UniqueName: \"kubernetes.io/projected/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-api-access-fcvrk\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041123 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.041115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-data-volume\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041213 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.041138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041213 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.041106    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.041306 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.041221    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.041306 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.041160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-crio-socket\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.041306 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.041220    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.541203519 +0000 UTC m=+44.286699297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.041306 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.041289    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:13.54127176 +0000 UTC m=+44.286767546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.041503 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.041470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-data-volume\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.051397 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.051368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvrk\" (UniqueName: \"kubernetes.io/projected/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-api-access-fcvrk\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.130408 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.130380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.225304 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.225237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.326998 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.326958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.426999 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.426956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.544697 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.544638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.544801 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.544699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:13.544801 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.544773    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.544801 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.544792    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.544891 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.544828    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.54481314 +0000 UTC m=+45.290308915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:13.544891 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.544840    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.544834375 +0000 UTC m=+45.290330150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:13.745840 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.745813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:13.745840 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.745843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:13.745999 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.745947    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:13.745999 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.745970    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.745949897 +0000 UTC m=+52.491445672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:13.746063 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.746012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.745998539 +0000 UTC m=+52.491494333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:13.746103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:13.746065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:13.746143 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.746133    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:13.746173 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:13.746164    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.746156333 +0000 UTC m=+52.491652107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:14.540999 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.540960    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bf6tv_317d9c48-0c40-4699-ab85-78fe0f29a92a/dns-node-resolver/0.log"
Sep 16 18:45:14.552383 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.552353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:14.552553 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.552448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:14.552553 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:14.552496    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:14.552848 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:14.552823    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.552778754 +0000 UTC m=+47.298274530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:14.552970 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:14.552906    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:14.556098 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:14.553138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.55308479 +0000 UTC m=+47.298580581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:14.974128 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.974097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ggz4p"]
Sep 16 18:45:14.986856 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.986834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ggz4p"]
Sep 16 18:45:14.986961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.986916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:14.989459 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.989440    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:45:14.989578 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.989464    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7vrh2\""
Sep 16 18:45:14.990246 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.990226    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:45:14.990345 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.990263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:45:14.990414 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:14.990369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:45:15.054835 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.054807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.156744 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.156715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.157754 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.157735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/460de112-e736-4149-a466-d96d6fa1f3c6-signing-key\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.157814 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.157790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/460de112-e736-4149-a466-d96d6fa1f3c6-signing-cabundle\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.157924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.157911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j4v\" (UniqueName: \"kubernetes.io/projected/460de112-e736-4149-a466-d96d6fa1f3c6-kube-api-access-76j4v\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.258323 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.258268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j4v\" (UniqueName: \"kubernetes.io/projected/460de112-e736-4149-a466-d96d6fa1f3c6-kube-api-access-76j4v\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.258323 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.258305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/460de112-e736-4149-a466-d96d6fa1f3c6-signing-key\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.258500 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.258485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/460de112-e736-4149-a466-d96d6fa1f3c6-signing-cabundle\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.259046 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.259028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/460de112-e736-4149-a466-d96d6fa1f3c6-signing-cabundle\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.260790 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.260773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/460de112-e736-4149-a466-d96d6fa1f3c6-signing-key\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.267709 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.267688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j4v\" (UniqueName: \"kubernetes.io/projected/460de112-e736-4149-a466-d96d6fa1f3c6-kube-api-access-76j4v\") pod \"service-ca-568869fdc5-ggz4p\" (UID: \"460de112-e736-4149-a466-d96d6fa1f3c6\") " pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.295825 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.295805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ggz4p"
Sep 16 18:45:15.587409 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.587329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ggz4p"]
Sep 16 18:45:15.591917 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:15.591890    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460de112_e736_4149_a466_d96d6fa1f3c6.slice/crio-a7d8f535b04bf1aa8c0fae08fb3ddd2a2d127190d265c776969cbd642d7bdd2a WatchSource:0}: Error finding container a7d8f535b04bf1aa8c0fae08fb3ddd2a2d127190d265c776969cbd642d7bdd2a: Status 404 returned error can't find the container with id a7d8f535b04bf1aa8c0fae08fb3ddd2a2d127190d265c776969cbd642d7bdd2a
Sep 16 18:45:15.941817 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:15.941792    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zfpxr_e1506a9e-c134-411b-8a43-0ff0bd857d2c/node-ca/0.log"
Sep 16 18:45:16.039675 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:16.039649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ggz4p" event={"ID":"460de112-e736-4149-a466-d96d6fa1f3c6","Type":"ContainerStarted","Data":"a7d8f535b04bf1aa8c0fae08fb3ddd2a2d127190d265c776969cbd642d7bdd2a"}
Sep 16 18:45:16.073971 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:16.073950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.342527 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:16.342441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2dvs4_53030e7e-f330-4608-9405-184801d5a706/serve-healthcheck-canary/0.log"
Sep 16 18:45:16.569839 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:16.569762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:16.570010 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:16.569868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:16.570010 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:16.569921    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:16.570010 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:16.569993    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:16.570140 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:16.570015    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.569978835 +0000 UTC m=+51.315474615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:16.570140 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:16.570046    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:20.570027879 +0000 UTC m=+51.315523670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:19.048448 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:19.048416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ggz4p" event={"ID":"460de112-e736-4149-a466-d96d6fa1f3c6","Type":"ContainerStarted","Data":"f532599afa33abf4f78457396faa5e4762e6e258cedceaacb27355bd39c4e6ec"}
Sep 16 18:45:19.070233 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:19.070183    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-ggz4p" podStartSLOduration=2.643662573 podStartE2EDuration="5.070167095s" totalImagesPullingTime="2.426504522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:19.069549259 +0000 UTC m=+49.815045056" watchObservedRunningTime="2026-09-16 18:45:19.070167095 +0000 UTC m=+49.815662891"
Sep 16 18:45:19.679815 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:19.679769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:19.899003 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:19.898957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:19.899161 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:19.899138    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:19.899220 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:19.899210    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls podName:475f96b1-e7c0-40b7-8ba0-a49b9c652fe3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.89919374 +0000 UTC m=+66.644689516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls") pod "dns-default-n8jt7" (UID: "475f96b1-e7c0-40b7-8ba0-a49b9c652fe3") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:20.165572 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.165538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:20.279867 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.279831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:20.379724 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.379691    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:20.605825 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.605723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:20.606031 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.605826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:20.606031 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:20.605892    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:20.606031 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:20.605908    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:20.606031 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:20.605971    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.605950715 +0000 UTC m=+59.351446490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:20.606031 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:20.606004    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.605979625 +0000 UTC m=+59.351475401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:20.753178 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:20.753148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.814582 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:21.814546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:21.814582 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:21.814583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:21.814686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:21.814775    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.814757162 +0000 UTC m=+68.560252937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:21.814797    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:21.814847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs podName:d23aa9cd-9131-483a-994e-483cbec4153f nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.814831236 +0000 UTC m=+68.560327012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs") pod "router-default-6469ffddb8-xx5ck" (UID: "d23aa9cd-9131-483a-994e-483cbec4153f") : secret "router-metrics-certs-default" not found
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:21.814861    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:21.815144 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:21.814888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls podName:ab59f219-af81-45d3-a712-d04595c124d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:37.814880365 +0000 UTC m=+68.560376142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-xkdm4" (UID: "ab59f219-af81-45d3-a712-d04595c124d7") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:21.961088 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:21.961056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:22.128149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:22.128099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:22.318865 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:22.318822    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:22.505275 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:22.505189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:22.712461 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:22.712430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.607534 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:23.607498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.811373 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:23.811332    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.912119 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:23.912093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.100232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:24.100211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.291294 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:24.291231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.492801 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:24.492777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.565340 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:24.565277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:24.698844 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:24.698818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:25.890474 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:25.890441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:25.993288 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:25.993259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.121176 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:26.121142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.303574 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:26.303498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.379136 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:26.379112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.495182 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:26.495155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.753335 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:26.753309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:27.686428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:27.686397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:27.751643 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:27.751613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.089675 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.089581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.286563 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.286529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.496203 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.496174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.663848 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.663810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:28.664035 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.663881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:28.664035 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:28.663955    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:28.664153 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:45:28.664059    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls podName:0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.664045086 +0000 UTC m=+75.409540861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fshll" (UID: "0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:28.664325 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.664308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:28.899325 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.899298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.988015 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:28.987978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqmkz"
Sep 16 18:45:29.494957 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:29.494925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:29.709012 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:29.708963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:30.110253 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:30.110206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:30.493933 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:30.493904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:30.690500 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:30.690469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:31.848011 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:31.847958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.514831 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.514798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:35.514831 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.514834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:35.517736 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.517716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:35.517806 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.517716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:35.528047 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.528018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d79c774c-df1c-4c16-bb55-a4a481ab692b-metrics-certs\") pod \"network-metrics-daemon-svtlg\" (UID: \"d79c774c-df1c-4c16-bb55-a4a481ab692b\") " pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:35.528149 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.528056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9f788b9-e314-4ef7-ab91-cf007fc45061-original-pull-secret\") pod \"global-pull-secret-syncer-6nq5t\" (UID: \"e9f788b9-e314-4ef7-ab91-cf007fc45061\") " pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:35.615796 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.615761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:35.618364 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.618349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.628827 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.628807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.639222 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.639203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4k2\" (UniqueName: \"kubernetes.io/projected/a62f8bbe-eca2-454e-9254-cae6e118f310-kube-api-access-nf4k2\") pod \"network-check-target-h44vj\" (UID: \"a62f8bbe-eca2-454e-9254-cae6e118f310\") " pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:35.753063 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.753032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rk9xq\""
Sep 16 18:45:35.757884 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.757864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvlxk\""
Sep 16 18:45:35.760584 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.760571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:35.760699 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.760685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nq5t"
Sep 16 18:45:35.766824 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.766785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svtlg"
Sep 16 18:45:35.918318 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.918284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:35.920484 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:35.920459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/475f96b1-e7c0-40b7-8ba0-a49b9c652fe3-metrics-tls\") pod \"dns-default-n8jt7\" (UID: \"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3\") " pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:36.056724 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.056647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h44vj"]
Sep 16 18:45:36.060144 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:36.060122    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62f8bbe_eca2_454e_9254_cae6e118f310.slice/crio-6c0cb4c9c3915bd33432a0553c5075a6b663c6526f3433f29ff78bbf82b716b8 WatchSource:0}: Error finding container 6c0cb4c9c3915bd33432a0553c5075a6b663c6526f3433f29ff78bbf82b716b8: Status 404 returned error can't find the container with id 6c0cb4c9c3915bd33432a0553c5075a6b663c6526f3433f29ff78bbf82b716b8
Sep 16 18:45:36.087114 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.087095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nq5t"]
Sep 16 18:45:36.091220 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:36.090510    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f788b9_e314_4ef7_ab91_cf007fc45061.slice/crio-5bb4e21c5e287a6f426bfb5e0a1b6aa5909ee28be67cebf1ed3c37bece8ba72f WatchSource:0}: Error finding container 5bb4e21c5e287a6f426bfb5e0a1b6aa5909ee28be67cebf1ed3c37bece8ba72f: Status 404 returned error can't find the container with id 5bb4e21c5e287a6f426bfb5e0a1b6aa5909ee28be67cebf1ed3c37bece8ba72f
Sep 16 18:45:36.093279 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.093252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h44vj" event={"ID":"a62f8bbe-eca2-454e-9254-cae6e118f310","Type":"ContainerStarted","Data":"6c0cb4c9c3915bd33432a0553c5075a6b663c6526f3433f29ff78bbf82b716b8"}
Sep 16 18:45:36.111962 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.111942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svtlg"]
Sep 16 18:45:36.114268 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:36.114248    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79c774c_df1c_4c16_bb55_a4a481ab692b.slice/crio-32a547496e175447a8a89ad056f121a16a963af5373036fcb229d88467e0c034 WatchSource:0}: Error finding container 32a547496e175447a8a89ad056f121a16a963af5373036fcb229d88467e0c034: Status 404 returned error can't find the container with id 32a547496e175447a8a89ad056f121a16a963af5373036fcb229d88467e0c034
Sep 16 18:45:36.201829 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.201807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7kfw\""
Sep 16 18:45:36.209777 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.209761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:36.504421 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.504393    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8jt7"]
Sep 16 18:45:36.507614 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:36.507591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475f96b1_e7c0_40b7_8ba0_a49b9c652fe3.slice/crio-32134abc3b22e23d2d8bd1817b2cba9d959d652035e17c2df30496f886c6ccca WatchSource:0}: Error finding container 32134abc3b22e23d2d8bd1817b2cba9d959d652035e17c2df30496f886c6ccca: Status 404 returned error can't find the container with id 32134abc3b22e23d2d8bd1817b2cba9d959d652035e17c2df30496f886c6ccca
Sep 16 18:45:36.616877 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.616844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:36.687878 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:36.687850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:37.109545 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.109506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jt7" event={"ID":"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3","Type":"ContainerStarted","Data":"32134abc3b22e23d2d8bd1817b2cba9d959d652035e17c2df30496f886c6ccca"}
Sep 16 18:45:37.111266 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.111238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svtlg" event={"ID":"d79c774c-df1c-4c16-bb55-a4a481ab692b","Type":"ContainerStarted","Data":"32a547496e175447a8a89ad056f121a16a963af5373036fcb229d88467e0c034"}
Sep 16 18:45:37.113346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.113311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nq5t" event={"ID":"e9f788b9-e314-4ef7-ab91-cf007fc45061","Type":"ContainerStarted","Data":"5bb4e21c5e287a6f426bfb5e0a1b6aa5909ee28be67cebf1ed3c37bece8ba72f"}
Sep 16 18:45:37.835223 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.835189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:37.835669 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.835238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:37.835669 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.835317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:37.835932 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.835903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23aa9cd-9131-483a-994e-483cbec4153f-service-ca-bundle\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:37.838767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.838741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab59f219-af81-45d3-a712-d04595c124d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-xkdm4\" (UID: \"ab59f219-af81-45d3-a712-d04595c124d7\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:37.842181 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:37.842155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23aa9cd-9131-483a-994e-483cbec4153f-metrics-certs\") pod \"router-default-6469ffddb8-xx5ck\" (UID: \"d23aa9cd-9131-483a-994e-483cbec4153f\") " pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:38.080954 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:38.080875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-drg7t\""
Sep 16 18:45:38.087374 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:38.087342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8hgf6\""
Sep 16 18:45:38.087531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:38.087484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"
Sep 16 18:45:38.095615 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:38.095590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:38.118554 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:38.118403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svtlg" event={"ID":"d79c774c-df1c-4c16-bb55-a4a481ab692b","Type":"ContainerStarted","Data":"9fcb04994a9aecba3bac0c610d10b92821c4b9573e8e54b9df17362c03b13e56"}
Sep 16 18:45:39.147269 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.147240    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"]
Sep 16 18:45:39.149428 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.149391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.153056 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.153032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:45:39.153056 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.153052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:45:39.153302 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.153281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:45:39.155512 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.155490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:45:39.170518 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.170498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"]
Sep 16 18:45:39.247278 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.247247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.247437 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.247291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xh2g\" (UniqueName: \"kubernetes.io/projected/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-kube-api-access-2xh2g\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.247437 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.247368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-tmp\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.261238 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.261183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:39.262449 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.262431    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59ccf65b59-t67xl"]
Sep 16 18:45:39.264508 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.264487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.278094 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.278076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rd9rd\""
Sep 16 18:45:39.278378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.278364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:39.278857 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.278700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:39.280649 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.280633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:39.294651 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.294634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59ccf65b59-t67xl"]
Sep 16 18:45:39.297522 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.297503    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:39.348367 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-tls\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348502 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xh2g\" (UniqueName: \"kubernetes.io/projected/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-kube-api-access-2xh2g\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.348502 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.348502 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-certificates\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348502 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-trusted-ca\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348709 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-ca-trust-extracted\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348709 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-installation-pull-secrets\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348709 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-image-registry-private-configuration\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348845 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55g8r\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-kube-api-access-55g8r\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.348845 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-tmp\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.348845 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.348815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-bound-sa-token\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.349153 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.349134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-tmp\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.351193 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.351175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.357380 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.357356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xh2g\" (UniqueName: \"kubernetes.io/projected/fb9e43a2-16a0-4668-9bd5-3b6b39577b73-kube-api-access-2xh2g\") pod \"klusterlet-addon-workmgr-b44d948b4-rzn99\" (UID: \"fb9e43a2-16a0-4668-9bd5-3b6b39577b73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.378796 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.378772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:39.449351 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-certificates\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449351 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-trusted-ca\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-ca-trust-extracted\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-installation-pull-secrets\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-image-registry-private-configuration\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55g8r\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-kube-api-access-55g8r\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-bound-sa-token\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-tls\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.449904 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.449869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-ca-trust-extracted\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.450248 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.450223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-certificates\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.450488 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.450439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-trusted-ca\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.452401 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.452374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-installation-pull-secrets\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.452543 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.452519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-image-registry-private-configuration\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.452618 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.452599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-registry-tls\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.459622 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.459601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-bound-sa-token\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.459725 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.459713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:39.460217 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.460188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55g8r\" (UniqueName: \"kubernetes.io/projected/988fe7cd-4b38-44b3-981a-0bfa8dcffd8a-kube-api-access-55g8r\") pod \"image-registry-59ccf65b59-t67xl\" (UID: \"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a\") " pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:39.575355 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:39.575320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:40.270664 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:40.270636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:40.371534 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:40.371500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.320083 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:41.320006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4"]
Sep 16 18:45:41.322204 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:41.321773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"]
Sep 16 18:45:41.322838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:41.322797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59ccf65b59-t67xl"]
Sep 16 18:45:41.324952 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:41.324923    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab59f219_af81_45d3_a712_d04595c124d7.slice/crio-1f5cbe656a3fb1a18efdfc5d8f6ba3d7e565a5dd356158f06a002eaf5c1124b8 WatchSource:0}: Error finding container 1f5cbe656a3fb1a18efdfc5d8f6ba3d7e565a5dd356158f06a002eaf5c1124b8: Status 404 returned error can't find the container with id 1f5cbe656a3fb1a18efdfc5d8f6ba3d7e565a5dd356158f06a002eaf5c1124b8
Sep 16 18:45:41.327539 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:41.327398    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9e43a2_16a0_4668_9bd5_3b6b39577b73.slice/crio-d79e244620bd5aa3c5b4c91fae2619ac50ddc74172d19d3f672335da46152ca2 WatchSource:0}: Error finding container d79e244620bd5aa3c5b4c91fae2619ac50ddc74172d19d3f672335da46152ca2: Status 404 returned error can't find the container with id d79e244620bd5aa3c5b4c91fae2619ac50ddc74172d19d3f672335da46152ca2
Sep 16 18:45:41.364069 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:41.363966    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6469ffddb8-xx5ck"]
Sep 16 18:45:41.368254 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:41.368226    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23aa9cd_9131_483a_994e_483cbec4153f.slice/crio-4d68dee9486bac8cc7757274860d8aaf7f4c156962311ed47724f6a29f880a56 WatchSource:0}: Error finding container 4d68dee9486bac8cc7757274860d8aaf7f4c156962311ed47724f6a29f880a56: Status 404 returned error can't find the container with id 4d68dee9486bac8cc7757274860d8aaf7f4c156962311ed47724f6a29f880a56
Sep 16 18:45:42.137531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.137493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svtlg" event={"ID":"d79c774c-df1c-4c16-bb55-a4a481ab692b","Type":"ContainerStarted","Data":"a72751827bcea21f9dd26b182b697b79eee98cd33705631ea8624f898753049c"}
Sep 16 18:45:42.139489 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.139456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99" event={"ID":"fb9e43a2-16a0-4668-9bd5-3b6b39577b73","Type":"ContainerStarted","Data":"d79e244620bd5aa3c5b4c91fae2619ac50ddc74172d19d3f672335da46152ca2"}
Sep 16 18:45:42.141450 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.141420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h44vj" event={"ID":"a62f8bbe-eca2-454e-9254-cae6e118f310","Type":"ContainerStarted","Data":"0a2efaddc492d35969c1f207d3632d740653629ffcdee0bbb08b663b452615ea"}
Sep 16 18:45:42.141557 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.141481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:45:42.147184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.144089    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nq5t" event={"ID":"e9f788b9-e314-4ef7-ab91-cf007fc45061","Type":"ContainerStarted","Data":"401a9b3697c387589db472e1cfee1ee74c3aaf787f464879820b7a863c154809"}
Sep 16 18:45:42.147184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.146238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jt7" event={"ID":"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3","Type":"ContainerStarted","Data":"e9cffd2bd63604b0cca318ef30b3a663200491105d67f050d9db7a1fd633981c"}
Sep 16 18:45:42.147184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.146284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8jt7" event={"ID":"475f96b1-e7c0-40b7-8ba0-a49b9c652fe3","Type":"ContainerStarted","Data":"45abaa96ad0a47cc7301fb86f33a7af7e4d5e020ad2341f774caff6e98327a7f"}
Sep 16 18:45:42.147184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.146845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:42.149060 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.149022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6469ffddb8-xx5ck" event={"ID":"d23aa9cd-9131-483a-994e-483cbec4153f","Type":"ContainerStarted","Data":"7f9ae9786b2728391327eaba02ada50a53b052a7cab6d29540e0cbcd821ba6bf"}
Sep 16 18:45:42.149159 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.149081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6469ffddb8-xx5ck" event={"ID":"d23aa9cd-9131-483a-994e-483cbec4153f","Type":"ContainerStarted","Data":"4d68dee9486bac8cc7757274860d8aaf7f4c156962311ed47724f6a29f880a56"}
Sep 16 18:45:42.151217 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.151197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl" event={"ID":"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a","Type":"ContainerStarted","Data":"a3fcc5fb77967c8e99fffb0bb5dae14546ee82ca67342f2bc4a3a0b162944f68"}
Sep 16 18:45:42.151311 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.151234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl" event={"ID":"988fe7cd-4b38-44b3-981a-0bfa8dcffd8a","Type":"ContainerStarted","Data":"9a9a5a55115a5d701d052aa33f2dbe73afeafd40be1d4e4d7f91ed64aa6b6bd2"}
Sep 16 18:45:42.151311 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.151282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:45:42.152969 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.152948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4" event={"ID":"ab59f219-af81-45d3-a712-d04595c124d7","Type":"ContainerStarted","Data":"1f5cbe656a3fb1a18efdfc5d8f6ba3d7e565a5dd356158f06a002eaf5c1124b8"}
Sep 16 18:45:42.175965 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.174766    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl" podStartSLOduration=3.174750991 podStartE2EDuration="3.174750991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.174423547 +0000 UTC m=+72.919919345" watchObservedRunningTime="2026-09-16 18:45:42.174750991 +0000 UTC m=+72.920246791"
Sep 16 18:45:42.175965 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.175230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-svtlg" podStartSLOduration=70.529629007 podStartE2EDuration="1m12.175219382s" totalImagesPullingTime="1.645590375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.154751983 +0000 UTC m=+72.900247797" watchObservedRunningTime="2026-09-16 18:45:42.175219382 +0000 UTC m=+72.920715180"
Sep 16 18:45:42.210777 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.210727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n8jt7" podStartSLOduration=33.701944513 podStartE2EDuration="38.210710638s" totalImagesPullingTime="4.508766125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.192677589 +0000 UTC m=+72.938173396" watchObservedRunningTime="2026-09-16 18:45:42.210710638 +0000 UTC m=+72.956206435"
Sep 16 18:45:42.232032 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.230751    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6469ffddb8-xx5ck" podStartSLOduration=37.230719253 podStartE2EDuration="37.230719253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.230157922 +0000 UTC m=+72.975653720" watchObservedRunningTime="2026-09-16 18:45:42.230719253 +0000 UTC m=+72.976215065"
Sep 16 18:45:42.232032 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.231555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6nq5t" podStartSLOduration=66.278425709 podStartE2EDuration="1m11.231547653s" totalImagesPullingTime="4.953121944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.211081123 +0000 UTC m=+72.956576923" watchObservedRunningTime="2026-09-16 18:45:42.231547653 +0000 UTC m=+72.977043449"
Sep 16 18:45:42.249041 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.248929    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h44vj" podStartSLOduration=66.268475222 podStartE2EDuration="1m11.248911819s" totalImagesPullingTime="4.980436597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.247799632 +0000 UTC m=+72.993295429" watchObservedRunningTime="2026-09-16 18:45:42.248911819 +0000 UTC m=+72.994407617"
Sep 16 18:45:42.262435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.262412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:42.364388 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:42.364358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:43.096421 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:43.096385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:43.099435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:43.099409    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:43.159181 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:43.159120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:43.159812 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:43.159661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6469ffddb8-xx5ck"
Sep 16 18:45:44.690905 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:44.690868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:44.693482 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:44.693457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fshll\" (UID: \"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61\") " pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:44.952076 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:44.951972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8svg\""
Sep 16 18:45:44.959702 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:44.959669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fshll"
Sep 16 18:45:45.522082 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:45.521973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fshll"]
Sep 16 18:45:45.524600 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:45.524572    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6eb8bd_e1bd_4c5e_ac82_31d352b8da61.slice/crio-40c7ece5afb2cd030940d286dda6821f304b9e5dab0b982d94b29f2937a4317f WatchSource:0}: Error finding container 40c7ece5afb2cd030940d286dda6821f304b9e5dab0b982d94b29f2937a4317f: Status 404 returned error can't find the container with id 40c7ece5afb2cd030940d286dda6821f304b9e5dab0b982d94b29f2937a4317f
Sep 16 18:45:45.628640 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:45.628607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.776051 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:45.775961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.921305 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:45.921272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:46.014655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.014615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:46.172112 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.172076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99" event={"ID":"fb9e43a2-16a0-4668-9bd5-3b6b39577b73","Type":"ContainerStarted","Data":"372326d170a347106290d80ae57afefb4ce75b9a8ff35d316b547783e59163bb"}
Sep 16 18:45:46.172312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.172224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:46.173832 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.173803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fshll" event={"ID":"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61","Type":"ContainerStarted","Data":"d91de73f9b0be830d1fd2aa457c309bbce6142c16d1f076f8e1012fa105721d3"}
Sep 16 18:45:46.173961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.173859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fshll" event={"ID":"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61","Type":"ContainerStarted","Data":"40c7ece5afb2cd030940d286dda6821f304b9e5dab0b982d94b29f2937a4317f"}
Sep 16 18:45:46.174349 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.174327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99"
Sep 16 18:45:46.175415 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.175385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4" event={"ID":"ab59f219-af81-45d3-a712-d04595c124d7","Type":"ContainerStarted","Data":"163c51642889c536da592de5cee11a56312db879d0a9f439cf787b69a071ef6a"}
Sep 16 18:45:46.189477 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.189230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44d948b4-rzn99" podStartSLOduration=3.2244493260000002 podStartE2EDuration="7.189213546s" totalImagesPullingTime="3.96476422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.188914691 +0000 UTC m=+76.934410487" watchObservedRunningTime="2026-09-16 18:45:46.189213546 +0000 UTC m=+76.934709343"
Sep 16 18:45:46.221506 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:46.221464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-xkdm4" podStartSLOduration=37.304557353 podStartE2EDuration="41.221449582s" totalImagesPullingTime="3.916892229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.220882932 +0000 UTC m=+76.966378755" watchObservedRunningTime="2026-09-16 18:45:46.221449582 +0000 UTC m=+76.966945379"
Sep 16 18:45:47.182151 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:47.182113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fshll" event={"ID":"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61","Type":"ContainerStarted","Data":"13b1f73c9cb994efecd2dfc4c8a5eff7cc8eaff567bcc4b5598be876b1484d6f"}
Sep 16 18:45:47.964939 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:47.964920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.064836 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:48.064806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.148268 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:48.148238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.187324 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:48.187253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fshll" event={"ID":"0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61","Type":"ContainerStarted","Data":"d0e54c3d95fdc01f96fe03b4f701ca11d859fa4e9826b90db5739159f906bc19"}
Sep 16 18:45:48.208938 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:48.208899    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fshll" podStartSLOduration=33.919085221 podStartE2EDuration="36.208884904s" totalImagesPullingTime="2.289799683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:48.207628739 +0000 UTC m=+78.953124536" watchObservedRunningTime="2026-09-16 18:45:48.208884904 +0000 UTC m=+78.954380698"
Sep 16 18:45:49.935916 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.935884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:49.972251 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.972227    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"]
Sep 16 18:45:49.976281 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.976266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:49.978766 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.978747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:45:49.978844 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.978762    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:45:49.980029 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.979980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:45:49.980136 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.980047    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-442qv\""
Sep 16 18:45:49.984430 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:49.984408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"]
Sep 16 18:45:50.010050 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.010030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.030122 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.030098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409662f8-ba71-4cd2-9433-7529b74575eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.030219 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.030161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.030219 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.030195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.030315 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.030233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkq7\" (UniqueName: \"kubernetes.io/projected/409662f8-ba71-4cd2-9433-7529b74575eb-kube-api-access-jhkq7\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.083287 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.083252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.130729 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.130705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409662f8-ba71-4cd2-9433-7529b74575eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.130792 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.130747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.130792 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.130769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.130869 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.130800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkq7\" (UniqueName: \"kubernetes.io/projected/409662f8-ba71-4cd2-9433-7529b74575eb-kube-api-access-jhkq7\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.131319 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.131301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409662f8-ba71-4cd2-9433-7529b74575eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.133239 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.133222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.133414 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.133393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/409662f8-ba71-4cd2-9433-7529b74575eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.140268 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.140251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkq7\" (UniqueName: \"kubernetes.io/projected/409662f8-ba71-4cd2-9433-7529b74575eb-kube-api-access-jhkq7\") pod \"prometheus-operator-68bcd7f987-ghc6d\" (UID: \"409662f8-ba71-4cd2-9433-7529b74575eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.285208 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.285152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"
Sep 16 18:45:50.556783 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:50.556703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d"]
Sep 16 18:45:50.561239 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:50.561206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409662f8_ba71_4cd2_9433_7529b74575eb.slice/crio-326605cd8b2c704e24beb65aa3c8754dbd6b5a61dbf60a6d5fdf656253d9b72f WatchSource:0}: Error finding container 326605cd8b2c704e24beb65aa3c8754dbd6b5a61dbf60a6d5fdf656253d9b72f: Status 404 returned error can't find the container with id 326605cd8b2c704e24beb65aa3c8754dbd6b5a61dbf60a6d5fdf656253d9b72f
Sep 16 18:45:51.198518 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:51.198478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d" event={"ID":"409662f8-ba71-4cd2-9433-7529b74575eb","Type":"ContainerStarted","Data":"326605cd8b2c704e24beb65aa3c8754dbd6b5a61dbf60a6d5fdf656253d9b72f"}
Sep 16 18:45:52.082282 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:52.082187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:52.160644 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:52.160618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n8jt7"
Sep 16 18:45:52.204971 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:52.204936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d" event={"ID":"409662f8-ba71-4cd2-9433-7529b74575eb","Type":"ContainerStarted","Data":"a65b6d8fb2b17d25c82d7d0345ed893ab05dc62af15eb95fae668de40a86a437"}
Sep 16 18:45:52.205416 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:52.205015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d" event={"ID":"409662f8-ba71-4cd2-9433-7529b74575eb","Type":"ContainerStarted","Data":"ba7407fde0f70e34b58c76e06aea442f1c190df86b2f0d05ee9ad93368f4778e"}
Sep 16 18:45:52.225290 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:52.225243    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ghc6d" podStartSLOduration=2.010336043 podStartE2EDuration="3.225228325s" totalImagesPullingTime="1.214892282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:52.224029638 +0000 UTC m=+82.969525435" watchObservedRunningTime="2026-09-16 18:45:52.225228325 +0000 UTC m=+82.970724121"
Sep 16 18:45:54.071761 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.071729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.288707 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.288672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.336934 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.336850    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4h7wb"]
Sep 16 18:45:54.341656 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.341637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.344047 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.344025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:45:54.344327 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.344312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:45:54.344744 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.344726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:45:54.344854 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.344726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9k2g7\""
Sep 16 18:45:54.378459 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.378430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.456549 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.456505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.462075 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-metrics-client-ca\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462206 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-root\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462206 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77087871-2210-4cae-b997-e8571af46130-node-exporter-textfile\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-node-exporter-wtmp\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4z5p\" (UniqueName: \"kubernetes.io/projected/77087871-2210-4cae-b997-e8571af46130-kube-api-access-z4z5p\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-tls\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462507 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-node-exporter-accelerators-collector-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.462507 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.462351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-sys\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.554484 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.554458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.563115 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-root\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77087871-2210-4cae-b997-e8571af46130-node-exporter-textfile\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-node-exporter-wtmp\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z5p\" (UniqueName: \"kubernetes.io/projected/77087871-2210-4cae-b997-e8571af46130-kube-api-access-z4z5p\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-tls\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563539 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-node-exporter-accelerators-collector-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563539 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-sys\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563539 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-metrics-client-ca\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563539 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77087871-2210-4cae-b997-e8571af46130-node-exporter-textfile\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563827 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-root\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.563927 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-metrics-client-ca\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.564044 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.563961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-node-exporter-wtmp\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.564270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.564217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77087871-2210-4cae-b997-e8571af46130-sys\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.564858 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.564831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77087871-2210-4cae-b997-e8571af46130-node-exporter-accelerators-collector-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.566184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.566161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.566665 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.566643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77087871-2210-4cae-b997-e8571af46130-node-exporter-tls\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.572616 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.572586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4z5p\" (UniqueName: \"kubernetes.io/projected/77087871-2210-4cae-b997-e8571af46130-kube-api-access-z4z5p\") pod \"node-exporter-4h7wb\" (UID: \"77087871-2210-4cae-b997-e8571af46130\") " pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.651403 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.651385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4h7wb"
Sep 16 18:45:54.658232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.658206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.663351 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:54.663331    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77087871_2210_4cae_b997_e8571af46130.slice/crio-d71cfcb1403766720630882d629b0c4b2d2602ab2b22e49afc26ba8cb7bfcf1e WatchSource:0}: Error finding container d71cfcb1403766720630882d629b0c4b2d2602ab2b22e49afc26ba8cb7bfcf1e: Status 404 returned error can't find the container with id d71cfcb1403766720630882d629b0c4b2d2602ab2b22e49afc26ba8cb7bfcf1e
Sep 16 18:45:54.873314 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:54.873282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.049006 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:55.048912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.218795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:55.218760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4h7wb" event={"ID":"77087871-2210-4cae-b997-e8571af46130","Type":"ContainerStarted","Data":"d71cfcb1403766720630882d629b0c4b2d2602ab2b22e49afc26ba8cb7bfcf1e"}
Sep 16 18:45:55.418121 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:55.418089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.506244 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:55.506206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.743924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:55.743897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.224755 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:56.224726    2580 generic.go:356] "Generic (PLEG): container finished" podID="77087871-2210-4cae-b997-e8571af46130" containerID="1123a05daebd2b3de582dc3cd4fe27a0e584df83d2b5520973b40c4e8e474158" exitCode=0
Sep 16 18:45:56.225236 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:56.224781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4h7wb" event={"ID":"77087871-2210-4cae-b997-e8571af46130","Type":"ContainerDied","Data":"1123a05daebd2b3de582dc3cd4fe27a0e584df83d2b5520973b40c4e8e474158"}
Sep 16 18:45:57.051255 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.051223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.142284 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.142254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.236219 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.232498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4h7wb" event={"ID":"77087871-2210-4cae-b997-e8571af46130","Type":"ContainerStarted","Data":"b487715cad94d5f7505c4591a6e7f497a1ec9f7ba6b63c4e51d06343d85111bf"}
Sep 16 18:45:57.236219 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.232862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4h7wb" event={"ID":"77087871-2210-4cae-b997-e8571af46130","Type":"ContainerStarted","Data":"a5288ef79a0c60220a91e455d849472f683dc19c18f822cb392f9fe0a8758463"}
Sep 16 18:45:57.272365 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.272323    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4h7wb" podStartSLOduration=2.227498643 podStartE2EDuration="3.272309792s" totalImagesPullingTime="972.44364ms" totalInitContainerRuntime="72.367509ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.272062959 +0000 UTC m=+88.017558756" watchObservedRunningTime="2026-09-16 18:45:57.272309792 +0000 UTC m=+88.017805590"
Sep 16 18:45:57.307839 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.307778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.382670 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.382634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.451466 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.451438    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7668b47f48-7bpg2"]
Sep 16 18:45:57.455030 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.455008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.457666 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.457646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7v77edeamk6te\""
Sep 16 18:45:57.457795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.457646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:45:57.457795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.457773    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-28wqm\""
Sep 16 18:45:57.457905 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.457845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:45:57.458046 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.458028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:45:57.458128 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.458062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:45:57.458128 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.458089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:45:57.466944 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.466925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7668b47f48-7bpg2"]
Sep 16 18:45:57.470145 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.470124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.586105 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586105 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586105 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0029a21-2249-44fb-8737-a7b9f5032d6a-metrics-client-ca\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6tn\" (UniqueName: \"kubernetes.io/projected/d0029a21-2249-44fb-8737-a7b9f5032d6a-kube-api-access-9d6tn\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-grpc-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.586378 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.586249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.598301 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.598278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.686908 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.686879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687094 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.686921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687218 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687319 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687319 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687439 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0029a21-2249-44fb-8737-a7b9f5032d6a-metrics-client-ca\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687439 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6tn\" (UniqueName: \"kubernetes.io/projected/d0029a21-2249-44fb-8737-a7b9f5032d6a-kube-api-access-9d6tn\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.687439 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.687410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-grpc-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.688581 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.688556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0029a21-2249-44fb-8737-a7b9f5032d6a-metrics-client-ca\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.689881 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.689855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.690026 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.690005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.690346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.690324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.690399 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.690359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.690706 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.690689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-grpc-tls\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.690749 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.690705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0029a21-2249-44fb-8737-a7b9f5032d6a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.696064 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.696044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6tn\" (UniqueName: \"kubernetes.io/projected/d0029a21-2249-44fb-8737-a7b9f5032d6a-kube-api-access-9d6tn\") pod \"thanos-querier-7668b47f48-7bpg2\" (UID: \"d0029a21-2249-44fb-8737-a7b9f5032d6a\") " pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.764713 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.764691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:45:57.881161 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:57.881135    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:58.043824 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:58.043760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7668b47f48-7bpg2"]
Sep 16 18:45:58.046346 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:45:58.046319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0029a21_2249_44fb_8737_a7b9f5032d6a.slice/crio-b431c3eaaec169d0ac60090d7dbc9b71fa41d6613ea938f94641dfc6892b78aa WatchSource:0}: Error finding container b431c3eaaec169d0ac60090d7dbc9b71fa41d6613ea938f94641dfc6892b78aa: Status 404 returned error can't find the container with id b431c3eaaec169d0ac60090d7dbc9b71fa41d6613ea938f94641dfc6892b78aa
Sep 16 18:45:58.236833 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:58.236755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"b431c3eaaec169d0ac60090d7dbc9b71fa41d6613ea938f94641dfc6892b78aa"}
Sep 16 18:45:59.093099 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.093051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.106353 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.106313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"]
Sep 16 18:45:59.109652 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.109612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.112678 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.112643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gbrpr\""
Sep 16 18:45:59.112782 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.112722    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:45:59.119915 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.119519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"]
Sep 16 18:45:59.198412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.198374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a774927-7dad-4469-81b7-b3495fb9248d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.198583 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.198432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52tc\" (UniqueName: \"kubernetes.io/projected/9a774927-7dad-4469-81b7-b3495fb9248d-kube-api-access-f52tc\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.251767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.251741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.298780 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.298752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a774927-7dad-4469-81b7-b3495fb9248d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.298912 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.298795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52tc\" (UniqueName: \"kubernetes.io/projected/9a774927-7dad-4469-81b7-b3495fb9248d-kube-api-access-f52tc\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.301358 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.301338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a774927-7dad-4469-81b7-b3495fb9248d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.307259 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.307240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52tc\" (UniqueName: \"kubernetes.io/projected/9a774927-7dad-4469-81b7-b3495fb9248d-kube-api-access-f52tc\") pod \"monitoring-plugin-5d85778b9d-pmf4s\" (UID: \"9a774927-7dad-4469-81b7-b3495fb9248d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.425036 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.425003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:45:59.857193 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.857098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.940701 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:45:59.940643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.042561 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.042538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.208794 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.208767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"]
Sep 16 18:46:00.211781 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:46:00.211762    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a774927_7dad_4469_81b7_b3495fb9248d.slice/crio-778be35f5e270967df6b050d6adba5e21443717d568c279f4db7a77f54c78bf3 WatchSource:0}: Error finding container 778be35f5e270967df6b050d6adba5e21443717d568c279f4db7a77f54c78bf3: Status 404 returned error can't find the container with id 778be35f5e270967df6b050d6adba5e21443717d568c279f4db7a77f54c78bf3
Sep 16 18:46:00.240417 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.240396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.245939 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.245920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"a03391a7bef31bfa38cf6851e62c42961ea9c3945e79e7efacd5de6e4de1692e"}
Sep 16 18:46:00.246037 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.245977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"d812acca7998375d1f472108be32f4132eb9d231a2fbdb778968713760b312d0"}
Sep 16 18:46:00.246086 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.246044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"760edf6b2c2bbf59089d4286ade5ac433bcd23a5652e670dafe2abc77d1a43a4"}
Sep 16 18:46:00.247051 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.247034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s" event={"ID":"9a774927-7dad-4469-81b7-b3495fb9248d","Type":"ContainerStarted","Data":"778be35f5e270967df6b050d6adba5e21443717d568c279f4db7a77f54c78bf3"}
Sep 16 18:46:00.326178 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.326156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.381726 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.381708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.678566 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.678488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.772490 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:00.772452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.347267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:01.347243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.449454 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:01.449347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.548236 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:01.548203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.741358 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:01.741263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.257876 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.257846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s" event={"ID":"9a774927-7dad-4469-81b7-b3495fb9248d","Type":"ContainerStarted","Data":"e33fc88a293a8edc6fc2e56186a81c1e76d7ef1e231e699f0bc5aa5f7f48b5af"}
Sep 16 18:46:02.258108 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.258028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:46:02.261349 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.261319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"bedf76c94268a5bf2fe92fe9fb2dbd8046075169d5cb4653930ff88e3c016be4"}
Sep 16 18:46:02.261456 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.261377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"f5935fc7d632ee55ac228aadbdf869c05d6c996584f235b4e6c47af5ec1cb142"}
Sep 16 18:46:02.261456 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.261403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" event={"ID":"d0029a21-2249-44fb-8737-a7b9f5032d6a","Type":"ContainerStarted","Data":"ae4871b00d8fda429616bc1cc17e8b1ddee1ebae4af5dff20ad37c2798af7612"}
Sep 16 18:46:02.261548 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.261475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:46:02.263923 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.263905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s"
Sep 16 18:46:02.283001 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.282945    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pmf4s" podStartSLOduration=1.4413373520000001 podStartE2EDuration="3.282930004s" totalImagesPullingTime="1.841592652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.282255496 +0000 UTC m=+93.027751294" watchObservedRunningTime="2026-09-16 18:46:02.282930004 +0000 UTC m=+93.028425795"
Sep 16 18:46:02.323132 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.323035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2" podStartSLOduration=2.355048138 podStartE2EDuration="5.323019276s" totalImagesPullingTime="2.967971138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.32195922 +0000 UTC m=+93.067455017" watchObservedRunningTime="2026-09-16 18:46:02.323019276 +0000 UTC m=+93.068515071"
Sep 16 18:46:02.664377 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:02.664345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.162781 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:03.162755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59ccf65b59-t67xl"
Sep 16 18:46:03.338788 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:03.338760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.435034 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:03.434937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:04.017346 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:04.017308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:04.118699 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:04.118662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:06.305370 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:06.305341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:06.367628 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:06.367604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:06.608863 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:06.608805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:06.682785 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:06.682768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:08.271090 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:08.271066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7668b47f48-7bpg2"
Sep 16 18:46:08.949492 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:08.949462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.432577 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:10.432548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.531332 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:10.531310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.612449 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:10.612421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.161129 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:13.161101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h44vj"
Sep 16 18:46:13.172221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:13.172194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.274399 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:13.274374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.166451 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:14.166414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:14.241400 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:14.241374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.203531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.203497    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:46:20.205758 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.205743    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.209735 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.209716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:46:20.210225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:46:20.210314 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:46:20.210314 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt58p\""
Sep 16 18:46:20.210840 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:46:20.210840 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210833    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:46:20.210972 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.210873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:46:20.211258 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.211238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:46:20.216058 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.216042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:46:20.229731 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.229713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:46:20.365071 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwww\" (UniqueName: \"kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365221 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.365412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.365271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465684 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465684 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465684 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwww\" (UniqueName: \"kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465890 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465890 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465890 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.465890 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.465764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.466448 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.466418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.466646 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.466627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.466719 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.466688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.466756 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.466739    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.468389 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.468370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.468451 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.468440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.477355 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.477337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwww\" (UniqueName: \"kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww\") pod \"console-7c6bdbb967-vqfmk\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") " pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.515321 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.515302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:20.841380 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.841353    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:46:20.842190 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:46:20.842166    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd80f59a_143a_4a51_a7a2_3310cc8462a4.slice/crio-62394c5002cb7505935c44f33ad7dee156d07a53045b7cf1108b36bb09bd4295 WatchSource:0}: Error finding container 62394c5002cb7505935c44f33ad7dee156d07a53045b7cf1108b36bb09bd4295: Status 404 returned error can't find the container with id 62394c5002cb7505935c44f33ad7dee156d07a53045b7cf1108b36bb09bd4295
Sep 16 18:46:20.978717 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:20.978687    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-xkdm4_ab59f219-af81-45d3-a712-d04595c124d7/cluster-monitoring-operator/0.log"
Sep 16 18:46:21.330045 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:21.330006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bdbb967-vqfmk" event={"ID":"dd80f59a-143a-4a51-a7a2-3310cc8462a4","Type":"ContainerStarted","Data":"62394c5002cb7505935c44f33ad7dee156d07a53045b7cf1108b36bb09bd4295"}
Sep 16 18:46:21.716599 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:21.716563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.851356 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:21.851329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.969245 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:21.969167    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-pmf4s_9a774927-7dad-4469-81b7-b3495fb9248d/monitoring-plugin/0.log"
Sep 16 18:46:22.170288 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:22.170261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/init-textfile/0.log"
Sep 16 18:46:22.369682 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:22.369629    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/node-exporter/0.log"
Sep 16 18:46:22.569873 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:22.569848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/kube-rbac-proxy/0.log"
Sep 16 18:46:24.343953 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:24.343922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bdbb967-vqfmk" event={"ID":"dd80f59a-143a-4a51-a7a2-3310cc8462a4","Type":"ContainerStarted","Data":"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"}
Sep 16 18:46:24.371811 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:24.371765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c6bdbb967-vqfmk" podStartSLOduration=1.717270084 podStartE2EDuration="4.371750268s" totalImagesPullingTime="2.654480184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.371213059 +0000 UTC m=+115.116708870" watchObservedRunningTime="2026-09-16 18:46:24.371750268 +0000 UTC m=+115.117246066"
Sep 16 18:46:25.970382 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:25.970345    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ghc6d_409662f8-ba71-4cd2-9433-7529b74575eb/prometheus-operator/0.log"
Sep 16 18:46:26.174190 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:26.174159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ghc6d_409662f8-ba71-4cd2-9433-7529b74575eb/kube-rbac-proxy/0.log"
Sep 16 18:46:27.169501 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:27.169472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/thanos-query/0.log"
Sep 16 18:46:27.369527 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:27.369496    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-web/0.log"
Sep 16 18:46:27.568637 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:27.568569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy/0.log"
Sep 16 18:46:27.768184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:27.768162    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/prom-label-proxy/0.log"
Sep 16 18:46:27.969435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:27.969411    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-rules/0.log"
Sep 16 18:46:28.169610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:28.169582    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-metrics/0.log"
Sep 16 18:46:28.355432 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:28.355355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.415429 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:28.415408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.168483 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:29.168453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.278139 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:29.278112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.368117 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:29.368097    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6469ffddb8-xx5ck_d23aa9cd-9131-483a-994e-483cbec4153f/router/0.log"
Sep 16 18:46:29.569081 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:29.569009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2dvs4_53030e7e-f330-4608-9405-184801d5a706/serve-healthcheck-canary/0.log"
Sep 16 18:46:30.516010 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:30.515960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:30.516389 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:30.516046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:30.520953 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:30.520924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:31.183314 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:31.183283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.279257 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:31.279213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.382648 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:31.382624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:46:31.602955 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:31.602881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.673473 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:31.673447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.604441 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:32.604392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.702353 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:32.702328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.616103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:36.616074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.716397 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:36.716367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.305729 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:47.305690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.404606 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:47.404580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.460025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:47.460001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.635520 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:51.635490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.737030 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:46:51.736976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.606586 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:10.606545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.708076 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:10.708047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:16.199238 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:16.199201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:16.302554 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:16.302519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.654552 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:21.654462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.770809 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:21.770775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.216665 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:44.216604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.380953 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:44.380915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:48.928688 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:48.928644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.024023 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:49.023999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:55.430465 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:55.430429    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:47:55.627862 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:55.627832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:55.730883 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:47:55.730800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:09.502752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:09.502720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:09.604047 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:09.604020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:09.683019 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:09.682970    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:20.449738 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.449685    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c6bdbb967-vqfmk" podUID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" containerName="console" containerID="cri-o://1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2" gracePeriod=15
Sep 16 18:48:20.684156 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.684132    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:48:20.730243 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwww\" (UniqueName: \"kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730243 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert\") pod \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\" (UID: \"dd80f59a-143a-4a51-a7a2-3310cc8462a4\") "
Sep 16 18:48:20.730686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config" (OuterVolumeSpecName: "console-config") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.730766 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.730892 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.730965 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.730941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.732742 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.732719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.732900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.732882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.733013 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.732969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww" (OuterVolumeSpecName: "kube-api-access-xpwww") pod "dd80f59a-143a-4a51-a7a2-3310cc8462a4" (UID: "dd80f59a-143a-4a51-a7a2-3310cc8462a4"). InnerVolumeSpecName "kube-api-access-xpwww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:20.766496 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.766473    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" containerID="1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2" exitCode=0
Sep 16 18:48:20.766607 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.766514    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bdbb967-vqfmk"
Sep 16 18:48:20.766607 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.766572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bdbb967-vqfmk" event={"ID":"dd80f59a-143a-4a51-a7a2-3310cc8462a4","Type":"ContainerDied","Data":"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"}
Sep 16 18:48:20.766676 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.766612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bdbb967-vqfmk" event={"ID":"dd80f59a-143a-4a51-a7a2-3310cc8462a4","Type":"ContainerDied","Data":"62394c5002cb7505935c44f33ad7dee156d07a53045b7cf1108b36bb09bd4295"}
Sep 16 18:48:20.766676 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.766632    2580 scope.go:117] "RemoveContainer" containerID="1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"
Sep 16 18:48:20.775359 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.775342    2580 scope.go:117] "RemoveContainer" containerID="1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"
Sep 16 18:48:20.775664 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:48:20.775646    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2\": container with ID starting with 1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2 not found: ID does not exist" containerID="1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"
Sep 16 18:48:20.775718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.775673    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2"} err="failed to get container status \"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2\": rpc error: code = NotFound desc = could not find container \"1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2\": container with ID starting with 1ccd377d826849093fc8267fb3bb9f82584d8bd1e54c0aff0cce79c518e0fad2 not found: ID does not exist"
Sep 16 18:48:20.791370 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.791348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:48:20.799285 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.799261    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6bdbb967-vqfmk"]
Sep 16 18:48:20.831478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831457    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-serving-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831476    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-oauth-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831486    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-oauth-serving-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831502    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwww\" (UniqueName: \"kubernetes.io/projected/dd80f59a-143a-4a51-a7a2-3310cc8462a4-kube-api-access-xpwww\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831513    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-console-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831521    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-service-ca\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.831610 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:20.831528    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd80f59a-143a-4a51-a7a2-3310cc8462a4-trusted-ca-bundle\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:48:21.743554 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:21.743523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" path="/var/lib/kubelet/pods/dd80f59a-143a-4a51-a7a2-3310cc8462a4/volumes"
Sep 16 18:48:33.117838 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:33.117782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:33.276911 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:33.276852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:39.432036 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:39.432005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:39.707760 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:39.707676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:39.916484 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:39.916438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.123119 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.123093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.225184 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.225155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.290004 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.289960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pzqv4"]
Sep 16 18:48:40.290251 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.290239    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" containerName="console"
Sep 16 18:48:40.290293 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.290253    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" containerName="console"
Sep 16 18:48:40.290326 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.290294    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd80f59a-143a-4a51-a7a2-3310cc8462a4" containerName="console"
Sep 16 18:48:40.294570 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.294556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.298112 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.298092    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:48:40.298112 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.298106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-876kh\""
Sep 16 18:48:40.298309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.298167    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:48:40.298309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.298187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:48:40.298309 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.298214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:48:40.303055 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.303036    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pzqv4"]
Sep 16 18:48:40.311865 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.311832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.370357 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.370326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8jc\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-kube-api-access-nf8jc\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.370470 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.370363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-certificates\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.374365 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.374302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.471241 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.471211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8jc\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-kube-api-access-nf8jc\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.471661 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.471248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-certificates\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.474108 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.474083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-certificates\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.479901 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.479880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8jc\" (UniqueName: \"kubernetes.io/projected/049fa6ec-737b-428b-8d45-d3bfa527d5ce-kube-api-access-nf8jc\") pod \"keda-admission-6c4d864b86-pzqv4\" (UID: \"049fa6ec-737b-428b-8d45-d3bfa527d5ce\") " pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.604435 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.604400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:40.626680 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.626609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.811753 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.811724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.880145 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.879937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pzqv4"]
Sep 16 18:48:40.882814 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:48:40.882789    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049fa6ec_737b_428b_8d45_d3bfa527d5ce.slice/crio-752eece83675830db5b4ce7cdd357532459105a81bef6db6a394f86e2ce3a601 WatchSource:0}: Error finding container 752eece83675830db5b4ce7cdd357532459105a81bef6db6a394f86e2ce3a601: Status 404 returned error can't find the container with id 752eece83675830db5b4ce7cdd357532459105a81bef6db6a394f86e2ce3a601
Sep 16 18:48:40.967335 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:40.967297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:41.068752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:41.068698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:41.838724 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:41.838685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4" event={"ID":"049fa6ec-737b-428b-8d45-d3bfa527d5ce","Type":"ContainerStarted","Data":"752eece83675830db5b4ce7cdd357532459105a81bef6db6a394f86e2ce3a601"}
Sep 16 18:48:43.847275 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:43.847193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4" event={"ID":"049fa6ec-737b-428b-8d45-d3bfa527d5ce","Type":"ContainerStarted","Data":"b79be210e6e5bc79cf5d325df41f3d65bad9f2a0604cd26c261db677962a1430"}
Sep 16 18:48:43.847632 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:43.847308    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:48:43.866291 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:48:43.866250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4" podStartSLOduration=1.38620856 podStartE2EDuration="3.866236946s" totalImagesPullingTime="2.480028386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:43.864951386 +0000 UTC m=+254.610447183" watchObservedRunningTime="2026-09-16 18:48:43.866236946 +0000 UTC m=+254.611732742"
Sep 16 18:49:04.854116 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:04.854085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-pzqv4"
Sep 16 18:49:22.231268 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:22.231232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:28.545118 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:28.545085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:29.623060 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:29.623032    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:30.790960 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:30.790925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:30.888289 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:30.888256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:46.396045 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:46.396004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.648844 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:49:47.648804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:01.588972 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.588915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:01.773852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.773818    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:01.823313 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.823281    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"]
Sep 16 18:50:01.826481 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.826460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.830482 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.830461    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:50:01.830595 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.830464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:50:01.830595 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.830464    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-tbcs4\""
Sep 16 18:50:01.830595 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.830545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:50:01.853676 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.853613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"]
Sep 16 18:50:01.869324 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.869290    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vgf7s"]
Sep 16 18:50:01.872320 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.872305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:01.873877 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.873849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpdh\" (UniqueName: \"kubernetes.io/projected/983871e1-653f-4287-a885-a2a9eed1c678-kube-api-access-zfpdh\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.874012 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.873958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983871e1-653f-4287-a885-a2a9eed1c678-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.875939 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.875921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:50:01.876079 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.876064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d9pmv\""
Sep 16 18:50:01.889009 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.888972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vgf7s"]
Sep 16 18:50:01.910666 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.910647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:01.975102 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.975067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983871e1-653f-4287-a885-a2a9eed1c678-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.975273 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.975112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpdh\" (UniqueName: \"kubernetes.io/projected/983871e1-653f-4287-a885-a2a9eed1c678-kube-api-access-zfpdh\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.975273 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.975147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp894\" (UniqueName: \"kubernetes.io/projected/1b9e5d8c-14c4-4e9b-b8a1-143500231825-kube-api-access-sp894\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:01.975273 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.975163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e5d8c-14c4-4e9b-b8a1-143500231825-data\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:01.977581 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.977557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983871e1-653f-4287-a885-a2a9eed1c678-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:01.985472 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:01.985443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpdh\" (UniqueName: \"kubernetes.io/projected/983871e1-653f-4287-a885-a2a9eed1c678-kube-api-access-zfpdh\") pod \"llmisvc-controller-manager-645f9c8fd6-jvt66\" (UID: \"983871e1-653f-4287-a885-a2a9eed1c678\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:02.055951 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.055922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:02.076091 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.076044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp894\" (UniqueName: \"kubernetes.io/projected/1b9e5d8c-14c4-4e9b-b8a1-143500231825-kube-api-access-sp894\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:02.076091 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.076092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e5d8c-14c4-4e9b-b8a1-143500231825-data\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:02.076442 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.076426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e5d8c-14c4-4e9b-b8a1-143500231825-data\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:02.095466 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.095430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp894\" (UniqueName: \"kubernetes.io/projected/1b9e5d8c-14c4-4e9b-b8a1-143500231825-kube-api-access-sp894\") pod \"seaweedfs-64568bcd49-vgf7s\" (UID: \"1b9e5d8c-14c4-4e9b-b8a1-143500231825\") " pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:02.136461 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.136431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:02.181520 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.181486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:02.414267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.414190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"]
Sep 16 18:50:02.417222 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:50:02.417195    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod983871e1_653f_4287_a885_a2a9eed1c678.slice/crio-4a7a351d4d26365a5a63d51f4f93b674144b22f8eeaba4078b47ecf43fe1e530 WatchSource:0}: Error finding container 4a7a351d4d26365a5a63d51f4f93b674144b22f8eeaba4078b47ecf43fe1e530: Status 404 returned error can't find the container with id 4a7a351d4d26365a5a63d51f4f93b674144b22f8eeaba4078b47ecf43fe1e530
Sep 16 18:50:02.418468 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.418451    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:02.468705 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:02.468672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vgf7s"]
Sep 16 18:50:02.471424 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:50:02.471389    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9e5d8c_14c4_4e9b_b8a1_143500231825.slice/crio-e42815d935ef78656b3ae99d7194882593248b3bc16b48aa01eadfbe8388351b WatchSource:0}: Error finding container e42815d935ef78656b3ae99d7194882593248b3bc16b48aa01eadfbe8388351b: Status 404 returned error can't find the container with id e42815d935ef78656b3ae99d7194882593248b3bc16b48aa01eadfbe8388351b
Sep 16 18:50:03.129205 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:03.129168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vgf7s" event={"ID":"1b9e5d8c-14c4-4e9b-b8a1-143500231825","Type":"ContainerStarted","Data":"e42815d935ef78656b3ae99d7194882593248b3bc16b48aa01eadfbe8388351b"}
Sep 16 18:50:03.130598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:03.130565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66" event={"ID":"983871e1-653f-4287-a885-a2a9eed1c678","Type":"ContainerStarted","Data":"4a7a351d4d26365a5a63d51f4f93b674144b22f8eeaba4078b47ecf43fe1e530"}
Sep 16 18:50:03.370106 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:03.370001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:03.456802 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:03.456695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:07.148936 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.148901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66" event={"ID":"983871e1-653f-4287-a885-a2a9eed1c678","Type":"ContainerStarted","Data":"a15efda50e5b8d7ad355bd5322f00de10d4df3aff0376321ad30dbfd88185507"}
Sep 16 18:50:07.149390 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.148963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:07.150412 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.150390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vgf7s" event={"ID":"1b9e5d8c-14c4-4e9b-b8a1-143500231825","Type":"ContainerStarted","Data":"e85259e1745536672be95b965f6d6aca0f7bc4bdb3123c3ab95a39876d35482d"}
Sep 16 18:50:07.150525 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.150510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:07.170432 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.170380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66" podStartSLOduration=2.483226143 podStartE2EDuration="6.170362402s" totalImagesPullingTime="3.687136259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:07.169605397 +0000 UTC m=+337.915101195" watchObservedRunningTime="2026-09-16 18:50:07.170362402 +0000 UTC m=+337.915858198"
Sep 16 18:50:07.190146 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:07.190078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vgf7s" podStartSLOduration=2.353244244 podStartE2EDuration="6.190054607s" totalImagesPullingTime="3.836810363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:07.189218719 +0000 UTC m=+337.934714518" watchObservedRunningTime="2026-09-16 18:50:07.190054607 +0000 UTC m=+337.935550405"
Sep 16 18:50:13.156614 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:13.156582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vgf7s"
Sep 16 18:50:38.157131 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:38.157061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvt66"
Sep 16 18:50:52.227686 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:50:52.227636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:06.852707 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:06.852678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:18.071720 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.071688    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64965fdb5c-6c4pd"]
Sep 16 18:51:18.073970 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.073948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.077538 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:51:18.077538 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:51:18.077726 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077550    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:51:18.077726 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:51:18.077726 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077539    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:51:18.077955 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:51:18.077955 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.077915    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:51:18.078338 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.078317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt58p\""
Sep 16 18:51:18.081702 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.081686    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:51:18.086587 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.086566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64965fdb5c-6c4pd"]
Sep 16 18:51:18.134260 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.134236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:18.136278 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136365 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-oauth-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136365 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-oauth-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136468 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdfj\" (UniqueName: \"kubernetes.io/projected/e15ba382-4195-4a10-9c3d-5462dfd990b7-kube-api-access-ltdfj\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136468 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136468 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-service-ca\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.136577 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.136472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-trusted-ca-bundle\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.202852 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.202828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:18.237154 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-oauth-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-oauth-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdfj\" (UniqueName: \"kubernetes.io/projected/e15ba382-4195-4a10-9c3d-5462dfd990b7-kube-api-access-ltdfj\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237267 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-service-ca\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.237478 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-trusted-ca-bundle\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.238012 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.238012 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-oauth-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.238012 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.237951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-service-ca\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.238292 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.238196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15ba382-4195-4a10-9c3d-5462dfd990b7-trusted-ca-bundle\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.239766 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.239747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-oauth-config\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.239880 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.239865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15ba382-4195-4a10-9c3d-5462dfd990b7-console-serving-cert\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.246067 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.246047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdfj\" (UniqueName: \"kubernetes.io/projected/e15ba382-4195-4a10-9c3d-5462dfd990b7-kube-api-access-ltdfj\") pod \"console-64965fdb5c-6c4pd\" (UID: \"e15ba382-4195-4a10-9c3d-5462dfd990b7\") " pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.384099 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.384078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:18.652811 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:18.652729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64965fdb5c-6c4pd"]
Sep 16 18:51:18.655891 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:51:18.655864    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15ba382_4195_4a10_9c3d_5462dfd990b7.slice/crio-f30c2cfef28894933c3e8cb6a2216fb799bad80fbddda08d70a71ff6ee620ea9 WatchSource:0}: Error finding container f30c2cfef28894933c3e8cb6a2216fb799bad80fbddda08d70a71ff6ee620ea9: Status 404 returned error can't find the container with id f30c2cfef28894933c3e8cb6a2216fb799bad80fbddda08d70a71ff6ee620ea9
Sep 16 18:51:19.423475 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:19.423443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64965fdb5c-6c4pd" event={"ID":"e15ba382-4195-4a10-9c3d-5462dfd990b7","Type":"ContainerStarted","Data":"13f72acd4f97c96662ebdaefc125d107707410a50ac7a31597b590369ab00672"}
Sep 16 18:51:19.423956 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:19.423496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64965fdb5c-6c4pd" event={"ID":"e15ba382-4195-4a10-9c3d-5462dfd990b7","Type":"ContainerStarted","Data":"f30c2cfef28894933c3e8cb6a2216fb799bad80fbddda08d70a71ff6ee620ea9"}
Sep 16 18:51:19.443457 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:19.443413    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64965fdb5c-6c4pd" podStartSLOduration=1.443395714 podStartE2EDuration="1.443395714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:19.442516175 +0000 UTC m=+410.188011972" watchObservedRunningTime="2026-09-16 18:51:19.443395714 +0000 UTC m=+410.188891494"
Sep 16 18:51:20.405008 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:20.404944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:20.507757 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:20.507709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:20.605646 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:20.605618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:20.707621 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:20.707540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:21.117115 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:21.117091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:21.923970 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:21.923932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:22.024495 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:22.024467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:28.384998 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.384953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:28.385461 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.385032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:28.389540 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.389522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:28.461790 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.461770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64965fdb5c-6c4pd"
Sep 16 18:51:28.647715 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.647638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:28.745117 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.745072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:28.804868 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:28.804844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:54.828679 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:54.828601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:54.898177 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:51:54.898145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:20.392465 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:52:20.392432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:55.380698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:52:55.380663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:55.480926 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:52:55.480902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:06.919203 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:06.919168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.065752 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.065725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.162400 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.162368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.172419 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.172361    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:53:07.176085 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.176068    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.178626 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:53:07.178746 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:53:07.178746 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178648    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:53:07.179182 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:53:07.179182 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:53:07.179182 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.179038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 18:53:07.179182 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.178926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:53:07.187265 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.187247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:53:07.247900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.247867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.247900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.247899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.248175 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.247933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.248175 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.247972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.248175 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.248019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87pjk\" (UniqueName: \"kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.268900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.268880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.349292 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87pjk\" (UniqueName: \"kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.349402 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.349402 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.349402 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.349402 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.349611 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:53:07.349473    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:53:07.349611 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:53:07.349544    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls podName:e9111c75-497d-45cd-872b-219fac17141c nodeName:}" failed. No retries permitted until 2026-09-16 18:53:07.849524012 +0000 UTC m=+518.595019805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls") pod "isvc-canary-rollback-predictor-665c868b97-vw7mk" (UID: "e9111c75-497d-45cd-872b-219fac17141c") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:53:07.349765 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.349748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.350165 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.350145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.350210 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.350182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.363609 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.363585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87pjk\" (UniqueName: \"kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.421903 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.421875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.521442 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.521385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:07.852440 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.852413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:07.854974 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:07.854955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vw7mk\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:08.088495 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:08.088422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:08.362148 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:08.362125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:53:08.364011 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:53:08.363970    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9111c75_497d_45cd_872b_219fac17141c.slice/crio-6f019a41dd3b5b0471380caa6e10c9a7bc2940c32c72cae738bc492de24970e9 WatchSource:0}: Error finding container 6f019a41dd3b5b0471380caa6e10c9a7bc2940c32c72cae738bc492de24970e9: Status 404 returned error can't find the container with id 6f019a41dd3b5b0471380caa6e10c9a7bc2940c32c72cae738bc492de24970e9
Sep 16 18:53:08.819172 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:08.819142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerStarted","Data":"6f019a41dd3b5b0471380caa6e10c9a7bc2940c32c72cae738bc492de24970e9"}
Sep 16 18:53:11.833270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:11.833191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerStarted","Data":"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"}
Sep 16 18:53:15.848111 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:15.848079    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9111c75-497d-45cd-872b-219fac17141c" containerID="257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c" exitCode=0
Sep 16 18:53:15.848512 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:15.848154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerDied","Data":"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"}
Sep 16 18:53:29.921811 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:29.921766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerStarted","Data":"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"}
Sep 16 18:53:31.932001 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:31.931953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerStarted","Data":"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"}
Sep 16 18:53:31.932361 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:31.932184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:31.932361 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:31.932215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:31.933540 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:31.933511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:31.953795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:31.953750    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podStartSLOduration=2.511473325 podStartE2EDuration="24.953736118s" totalImagesPullingTime="18.591359187s" totalInitContainerRuntime="3.850903606s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:31.951884079 +0000 UTC m=+542.697379874" watchObservedRunningTime="2026-09-16 18:53:31.953736118 +0000 UTC m=+542.699231915"
Sep 16 18:53:32.935605 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:32.935571    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:37.939901 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:37.939872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:53:37.940357 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:37.940333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:47.940354 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:47.940300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:53:57.940601 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:53:57.940559    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:07.940388 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:07.940347    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:17.941150 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:17.941112    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:27.940705 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:27.940662    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:37.941066 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:37.940599    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:54:47.940993 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:47.940952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:54:56.939527 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:54:56.939487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:01.966417 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:01.966384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.066663 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.066630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.073038 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.073012    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:55:02.076447 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.076432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.078722 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.078703    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2e185-predictor-serving-cert\""
Sep 16 18:55:02.079113 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.079096    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2e185-kube-rbac-proxy-sar-config\""
Sep 16 18:55:02.091344 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.091322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:55:02.122281 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.122253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hhw\" (UniqueName: \"kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.122444 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.122291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.122444 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.122322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.122444 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.122353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.122444 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.122388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.196286 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.196254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.221577 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.221497    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:55:02.223462 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.223440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hhw\" (UniqueName: \"kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.223547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.223473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.223547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.223498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.223547 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.223537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.223684 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.223575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.224045 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.224018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.224168 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.224152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.224368 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.224341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.225622 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.225606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.226261 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.226242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.227936 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.227921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2e185-predictor-serving-cert\""
Sep 16 18:55:02.228082 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.227975    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2e185-kube-rbac-proxy-sar-config\""
Sep 16 18:55:02.235560 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.235532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:55:02.237545 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.237524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hhw\" (UniqueName: \"kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw\") pod \"isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.269632 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.269590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.324778 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.324739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.324961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.324787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.324961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.324815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldf5\" (UniqueName: \"kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.324961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.324838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.324961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.324917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.387836 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.387793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:02.426204 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.426383 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.426383 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldf5\" (UniqueName: \"kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.426383 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.426383 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.426648 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:55:02.426385    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-serving-cert: secret "isvc-xgboost-graph-raw-2e185-predictor-serving-cert" not found
Sep 16 18:55:02.426648 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:55:02.426459    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls podName:656f7793-475e-4a3f-8731-a8b7a64d58dc nodeName:}" failed. No retries permitted until 2026-09-16 18:55:02.926437851 +0000 UTC m=+633.671933646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls") pod "isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc") : secret "isvc-xgboost-graph-raw-2e185-predictor-serving-cert" not found
Sep 16 18:55:02.426830 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.426808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.427110 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.427089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.427183 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.427089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.436581 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.436559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldf5\" (UniqueName: \"kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.579017 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.578910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:02.665465 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.665421    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:55:02.672076 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:55:02.672043    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb984e1a2_de74_4af2_850f_42736b1afca0.slice/crio-ede2a2848137d8b9a45556eedb65a3f766dc091a8ab762e5e4b4a6135a8abe7b WatchSource:0}: Error finding container ede2a2848137d8b9a45556eedb65a3f766dc091a8ab762e5e4b4a6135a8abe7b: Status 404 returned error can't find the container with id ede2a2848137d8b9a45556eedb65a3f766dc091a8ab762e5e4b4a6135a8abe7b
Sep 16 18:55:02.930900 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.930855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:02.933387 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:02.933355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") pod \"isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:03.151416 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:03.151378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:03.279166 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:03.279134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerStarted","Data":"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"}
Sep 16 18:55:03.279300 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:03.279190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerStarted","Data":"ede2a2848137d8b9a45556eedb65a3f766dc091a8ab762e5e4b4a6135a8abe7b"}
Sep 16 18:55:03.434670 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:03.434540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:55:03.437199 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:55:03.437168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656f7793_475e_4a3f_8731_a8b7a64d58dc.slice/crio-2c1e7b9872c09f36e4c4261ba44a41a0e531bf81b1637a1dbabea194833e50b2 WatchSource:0}: Error finding container 2c1e7b9872c09f36e4c4261ba44a41a0e531bf81b1637a1dbabea194833e50b2: Status 404 returned error can't find the container with id 2c1e7b9872c09f36e4c4261ba44a41a0e531bf81b1637a1dbabea194833e50b2
Sep 16 18:55:04.284710 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:04.284669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerStarted","Data":"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"}
Sep 16 18:55:04.285143 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:04.284742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerStarted","Data":"2c1e7b9872c09f36e4c4261ba44a41a0e531bf81b1637a1dbabea194833e50b2"}
Sep 16 18:55:07.298218 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:07.298184    2580 generic.go:356] "Generic (PLEG): container finished" podID="b984e1a2-de74-4af2-850f-42736b1afca0" containerID="e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c" exitCode=0
Sep 16 18:55:07.298731 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:07.298264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerDied","Data":"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"}
Sep 16 18:55:08.305415 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.305371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerStarted","Data":"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"}
Sep 16 18:55:08.305847 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.305461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerStarted","Data":"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"}
Sep 16 18:55:08.305847 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.305804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:08.305976 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.305953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:08.307206 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.307182    2580 generic.go:356] "Generic (PLEG): container finished" podID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerID="90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087" exitCode=0
Sep 16 18:55:08.307312 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.307226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerDied","Data":"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"}
Sep 16 18:55:08.307487 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.307466    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:08.308553 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.308539    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:08.327705 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:08.327661    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podStartSLOduration=2.401505368 podStartE2EDuration="6.327646871s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.926141503s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:08.325532773 +0000 UTC m=+639.071028575" watchObservedRunningTime="2026-09-16 18:55:08.327646871 +0000 UTC m=+639.073142667"
Sep 16 18:55:09.312899 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:09.312841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:14.317430 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:14.317397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:55:14.318001 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:14.317954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:24.318176 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:24.318139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:26.399006 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:26.398954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerStarted","Data":"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"}
Sep 16 18:55:26.399469 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:26.399036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerStarted","Data":"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"}
Sep 16 18:55:26.399469 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:26.399219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:27.405317 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:27.405286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:27.406431 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:27.406395    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:28.409454 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:28.409417    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:33.415239 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:33.415207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:55:33.415749 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:33.415721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:33.437789 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:33.437743    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podStartSLOduration=10.572653368 podStartE2EDuration="31.437727594s" totalImagesPullingTime="17.043674578s" totalInitContainerRuntime="3.821399648s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:26.423640969 +0000 UTC m=+657.169136766" watchObservedRunningTime="2026-09-16 18:55:33.437727594 +0000 UTC m=+664.183223390"
Sep 16 18:55:34.318047 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:34.317980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:43.416010 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:43.415955    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:44.318903 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:44.318864    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:55:53.416436 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:53.416399    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:55:54.317831 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:55:54.317794    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:56:03.416517 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:03.416472    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:56:04.318297 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:04.318260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:56:13.416199 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:13.416157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:56:14.318344 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:14.318299    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:56:23.416226 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:23.416149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:56:24.319021 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:24.318970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:56:27.006185 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.006152    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:56:27.006718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.006445    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" containerID="cri-o://fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf" gracePeriod=30
Sep 16 18:56:27.006718 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.006528    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kube-rbac-proxy" containerID="cri-o://b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341" gracePeriod=30
Sep 16 18:56:27.030031 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.030003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:27.661633 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.661600    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9111c75-497d-45cd-872b-219fac17141c" containerID="b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341" exitCode=2
Sep 16 18:56:27.661807 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.661677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerDied","Data":"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"}
Sep 16 18:56:27.936748 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.936655    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 16 18:56:27.940680 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:27.940653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 16 18:56:31.151278 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.151255    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:56:31.245630 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location\") pod \"e9111c75-497d-45cd-872b-219fac17141c\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") "
Sep 16 18:56:31.245781 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert\") pod \"e9111c75-497d-45cd-872b-219fac17141c\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") "
Sep 16 18:56:31.245781 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87pjk\" (UniqueName: \"kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk\") pod \"e9111c75-497d-45cd-872b-219fac17141c\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") "
Sep 16 18:56:31.245781 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config\") pod \"e9111c75-497d-45cd-872b-219fac17141c\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") "
Sep 16 18:56:31.245932 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") pod \"e9111c75-497d-45cd-872b-219fac17141c\" (UID: \"e9111c75-497d-45cd-872b-219fac17141c\") "
Sep 16 18:56:31.246003 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.245959    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9111c75-497d-45cd-872b-219fac17141c" (UID: "e9111c75-497d-45cd-872b-219fac17141c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:31.246120 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.246099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9111c75-497d-45cd-872b-219fac17141c" (UID: "e9111c75-497d-45cd-872b-219fac17141c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:31.246186 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.246114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9111c75-497d-45cd-872b-219fac17141c" (UID: "e9111c75-497d-45cd-872b-219fac17141c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:31.248161 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.248136    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9111c75-497d-45cd-872b-219fac17141c" (UID: "e9111c75-497d-45cd-872b-219fac17141c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:31.248161 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.248153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk" (OuterVolumeSpecName: "kube-api-access-87pjk") pod "e9111c75-497d-45cd-872b-219fac17141c" (UID: "e9111c75-497d-45cd-872b-219fac17141c"). InnerVolumeSpecName "kube-api-access-87pjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:31.347042 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.346950    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9111c75-497d-45cd-872b-219fac17141c-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.347042 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.346976    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.347042 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.347003    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87pjk\" (UniqueName: \"kubernetes.io/projected/e9111c75-497d-45cd-872b-219fac17141c-kube-api-access-87pjk\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.347042 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.347012    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9111c75-497d-45cd-872b-219fac17141c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.347042 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.347022    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9111c75-497d-45cd-872b-219fac17141c-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.678784 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.678749    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9111c75-497d-45cd-872b-219fac17141c" containerID="fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf" exitCode=0
Sep 16 18:56:31.678967 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.678812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerDied","Data":"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"}
Sep 16 18:56:31.678967 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.678822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"
Sep 16 18:56:31.678967 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.678882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk" event={"ID":"e9111c75-497d-45cd-872b-219fac17141c","Type":"ContainerDied","Data":"6f019a41dd3b5b0471380caa6e10c9a7bc2940c32c72cae738bc492de24970e9"}
Sep 16 18:56:31.678967 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.678908    2580 scope.go:117] "RemoveContainer" containerID="b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"
Sep 16 18:56:31.687813 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.687794    2580 scope.go:117] "RemoveContainer" containerID="fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"
Sep 16 18:56:31.694969 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.694949    2580 scope.go:117] "RemoveContainer" containerID="257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"
Sep 16 18:56:31.701235 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.701215    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:56:31.702391 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.702375    2580 scope.go:117] "RemoveContainer" containerID="b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"
Sep 16 18:56:31.702644 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:31.702616    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341\": container with ID starting with b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341 not found: ID does not exist" containerID="b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"
Sep 16 18:56:31.702738 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.702643    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341"} err="failed to get container status \"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341\": rpc error: code = NotFound desc = could not find container \"b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341\": container with ID starting with b86015002905b2ce89e1c0155c0aea7cc22921cdcea6e008ce32cf879e99d341 not found: ID does not exist"
Sep 16 18:56:31.702738 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.702661    2580 scope.go:117] "RemoveContainer" containerID="fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"
Sep 16 18:56:31.702970 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:31.702938    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf\": container with ID starting with fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf not found: ID does not exist" containerID="fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"
Sep 16 18:56:31.703050 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.702972    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf"} err="failed to get container status \"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf\": rpc error: code = NotFound desc = could not find container \"fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf\": container with ID starting with fe8b434485e515b20010174e6b99b5aa6006a7c52042053fec0bcd18b93251bf not found: ID does not exist"
Sep 16 18:56:31.703050 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.703006    2580 scope.go:117] "RemoveContainer" containerID="257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"
Sep 16 18:56:31.703226 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:31.703206    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c\": container with ID starting with 257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c not found: ID does not exist" containerID="257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"
Sep 16 18:56:31.703315 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.703233    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c"} err="failed to get container status \"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c\": rpc error: code = NotFound desc = could not find container \"257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c\": container with ID starting with 257a81ed6b30e4d762e61749c882160e0a3e689bc8bdb71caecc8c42d69e251c not found: ID does not exist"
Sep 16 18:56:31.705123 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.705102    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vw7mk"]
Sep 16 18:56:31.744661 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:31.744635    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9111c75-497d-45cd-872b-219fac17141c" path="/var/lib/kubelet/pods/e9111c75-497d-45cd-872b-219fac17141c/volumes"
Sep 16 18:56:32.207499 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.207470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:32.254103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254071    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:56:32.254553 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254540    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container"
Sep 16 18:56:32.254602 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254557    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container"
Sep 16 18:56:32.254602 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="storage-initializer"
Sep 16 18:56:32.254602 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254596    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="storage-initializer"
Sep 16 18:56:32.254696 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kube-rbac-proxy"
Sep 16 18:56:32.254696 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kube-rbac-proxy"
Sep 16 18:56:32.254696 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kube-rbac-proxy"
Sep 16 18:56:32.254784 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.254698    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9111c75-497d-45cd-872b-219fac17141c" containerName="kserve-container"
Sep 16 18:56:32.258185 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.258166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.260439 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.260414    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:56:32.260893 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.260876    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:56:32.268264 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.268244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:56:32.290564 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.290541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:32.356775 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.356749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.356895 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.356778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ngd\" (UniqueName: \"kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.356895 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.356796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.356895 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.356817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.357050 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.356934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.414331 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.414307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:32.457425 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.457537 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ngd\" (UniqueName: \"kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.457537 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.457537 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.457699 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.457868 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.457849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.458268 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.458247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.458325 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.458252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.460025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.460006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.465560 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.465542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ngd\" (UniqueName: \"kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd\") pod \"isvc-canary-stop-predictor-7bb9c548b5-4txf9\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.569791 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.569760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:32.840558 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:32.840533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:56:32.842706 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:56:32.842683    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad02d13_005f_4a3d_86a5_8a26f3aac79e.slice/crio-3e5ba0d114307329f7b9a26c11881b544e3b5778efc34e506cfc4cd0099440cd WatchSource:0}: Error finding container 3e5ba0d114307329f7b9a26c11881b544e3b5778efc34e506cfc4cd0099440cd: Status 404 returned error can't find the container with id 3e5ba0d114307329f7b9a26c11881b544e3b5778efc34e506cfc4cd0099440cd
Sep 16 18:56:33.689968 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:33.689933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerStarted","Data":"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"}
Sep 16 18:56:33.690330 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:33.690006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerStarted","Data":"3e5ba0d114307329f7b9a26c11881b544e3b5778efc34e506cfc4cd0099440cd"}
Sep 16 18:56:36.703753 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:36.703664    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerID="69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd" exitCode=0
Sep 16 18:56:36.703753 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:36.703737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerDied","Data":"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"}
Sep 16 18:56:37.710604 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:37.710569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerStarted","Data":"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"}
Sep 16 18:56:37.711091 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:37.710666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerStarted","Data":"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"}
Sep 16 18:56:37.711091 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:37.710893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:37.733952 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:37.733907    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podStartSLOduration=2.235675464 podStartE2EDuration="5.733894111s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.498218647s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:37.730495628 +0000 UTC m=+728.475991424" watchObservedRunningTime="2026-09-16 18:56:37.733894111 +0000 UTC m=+728.479389962"
Sep 16 18:56:38.714673 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:38.714639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:38.716163 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:38.716131    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:56:39.718928 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:39.718885    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:56:42.108507 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:42.108471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:42.304741 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:42.304714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:44.724004 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:44.723959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:56:44.724511 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:44.724482    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:56:52.181186 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.181151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.275289 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.275260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:56:52.275531 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.275511    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" containerID="cri-o://f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51" gracePeriod=30
Sep 16 18:56:52.275645 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.275615    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kube-rbac-proxy" containerID="cri-o://efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54" gracePeriod=30
Sep 16 18:56:52.305607 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.305532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.322607 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.322581    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:56:52.322867 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.322819    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" containerID="cri-o://be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d" gracePeriod=30
Sep 16 18:56:52.322975 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.322869    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kube-rbac-proxy" containerID="cri-o://84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41" gracePeriod=30
Sep 16 18:56:52.405319 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.405291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.507652 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.507558    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.598023 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.597969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.749541 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.749447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.776642 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.776616    2580 generic.go:356] "Generic (PLEG): container finished" podID="b984e1a2-de74-4af2-850f-42736b1afca0" containerID="efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54" exitCode=2
Sep 16 18:56:52.776776 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.776693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerDied","Data":"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"}
Sep 16 18:56:52.778861 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.778842    2580 generic.go:356] "Generic (PLEG): container finished" podID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerID="84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41" exitCode=2
Sep 16 18:56:52.778954 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:52.778883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerDied","Data":"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"}
Sep 16 18:56:53.410467 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:53.410423    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 16 18:56:53.415741 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:53.415718    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 16 18:56:54.313699 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:54.313659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 16 18:56:54.318014 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:54.317973    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 16 18:56:54.724879 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:54.724835    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:56:55.952673 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:55.952638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:56.165465 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.165445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:56:56.242940 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.242864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bldf5\" (UniqueName: \"kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5\") pod \"656f7793-475e-4a3f-8731-a8b7a64d58dc\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") "
Sep 16 18:56:56.242940 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.242908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config\") pod \"656f7793-475e-4a3f-8731-a8b7a64d58dc\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") "
Sep 16 18:56:56.243277 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.243249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "656f7793-475e-4a3f-8731-a8b7a64d58dc" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:56.244961 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.244944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5" (OuterVolumeSpecName: "kube-api-access-bldf5") pod "656f7793-475e-4a3f-8731-a8b7a64d58dc" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc"). InnerVolumeSpecName "kube-api-access-bldf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:56.343841 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.343812    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") pod \"656f7793-475e-4a3f-8731-a8b7a64d58dc\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") "
Sep 16 18:56:56.343841 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.343842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert\") pod \"656f7793-475e-4a3f-8731-a8b7a64d58dc\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") "
Sep 16 18:56:56.344085 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.343873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location\") pod \"656f7793-475e-4a3f-8731-a8b7a64d58dc\" (UID: \"656f7793-475e-4a3f-8731-a8b7a64d58dc\") "
Sep 16 18:56:56.344175 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.344125    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bldf5\" (UniqueName: \"kubernetes.io/projected/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-api-access-bldf5\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:56.344175 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.344144    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:56.344281 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.344189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "656f7793-475e-4a3f-8731-a8b7a64d58dc" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:56.344281 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.344253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "656f7793-475e-4a3f-8731-a8b7a64d58dc" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:56.345938 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.345917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "656f7793-475e-4a3f-8731-a8b7a64d58dc" (UID: "656f7793-475e-4a3f-8731-a8b7a64d58dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:56.445142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.445114    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656f7793-475e-4a3f-8731-a8b7a64d58dc-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:56.445142 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.445138    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656f7793-475e-4a3f-8731-a8b7a64d58dc-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:56.445300 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.445173    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656f7793-475e-4a3f-8731-a8b7a64d58dc-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:56.796839 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.796799    2580 generic.go:356] "Generic (PLEG): container finished" podID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerID="be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d" exitCode=0
Sep 16 18:56:56.797025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.796860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"
Sep 16 18:56:56.797025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.796883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerDied","Data":"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"}
Sep 16 18:56:56.797025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.796943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294" event={"ID":"656f7793-475e-4a3f-8731-a8b7a64d58dc","Type":"ContainerDied","Data":"2c1e7b9872c09f36e4c4261ba44a41a0e531bf81b1637a1dbabea194833e50b2"}
Sep 16 18:56:56.797025 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.796969    2580 scope.go:117] "RemoveContainer" containerID="84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"
Sep 16 18:56:56.807209 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.807190    2580 scope.go:117] "RemoveContainer" containerID="be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"
Sep 16 18:56:56.820519 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.820498    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:56:56.824472 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.824451    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2e185-predictor-64d7f6dbdf-ss294"]
Sep 16 18:56:56.850298 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.850282    2580 scope.go:117] "RemoveContainer" containerID="90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"
Sep 16 18:56:56.886179 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.886162    2580 scope.go:117] "RemoveContainer" containerID="84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"
Sep 16 18:56:56.886454 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:56.886434    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41\": container with ID starting with 84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41 not found: ID does not exist" containerID="84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"
Sep 16 18:56:56.886524 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.886458    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41"} err="failed to get container status \"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41\": rpc error: code = NotFound desc = could not find container \"84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41\": container with ID starting with 84819f9295e2b7148af1ee612c5a57587926b9e0ca45dfb1a83dd1ab30fc3c41 not found: ID does not exist"
Sep 16 18:56:56.886524 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.886475    2580 scope.go:117] "RemoveContainer" containerID="be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"
Sep 16 18:56:56.886759 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:56.886728    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d\": container with ID starting with be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d not found: ID does not exist" containerID="be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"
Sep 16 18:56:56.886797 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.886766    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d"} err="failed to get container status \"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d\": rpc error: code = NotFound desc = could not find container \"be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d\": container with ID starting with be39da329c148c4f06cc5afb8718f27fdda58fc953787d152d144173e577160d not found: ID does not exist"
Sep 16 18:56:56.886797 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.886783    2580 scope.go:117] "RemoveContainer" containerID="90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"
Sep 16 18:56:56.887086 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:56.887060    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087\": container with ID starting with 90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087 not found: ID does not exist" containerID="90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"
Sep 16 18:56:56.887179 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:56.887093    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087"} err="failed to get container status \"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087\": rpc error: code = NotFound desc = could not find container \"90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087\": container with ID starting with 90e1e2b66639f78bb2c124fb40634095844ea83fbda2bbd6e657f1d11efb5087 not found: ID does not exist"
Sep 16 18:56:57.010532 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.010499    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:56:57.048930 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.048861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert\") pod \"b984e1a2-de74-4af2-850f-42736b1afca0\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") "
Sep 16 18:56:57.048930 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.048909    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location\") pod \"b984e1a2-de74-4af2-850f-42736b1afca0\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") "
Sep 16 18:56:57.049144 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.048960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82hhw\" (UniqueName: \"kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw\") pod \"b984e1a2-de74-4af2-850f-42736b1afca0\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") "
Sep 16 18:56:57.049144 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls\") pod \"b984e1a2-de74-4af2-850f-42736b1afca0\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") "
Sep 16 18:56:57.049144 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config\") pod \"b984e1a2-de74-4af2-850f-42736b1afca0\" (UID: \"b984e1a2-de74-4af2-850f-42736b1afca0\") "
Sep 16 18:56:57.049299 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b984e1a2-de74-4af2-850f-42736b1afca0" (UID: "b984e1a2-de74-4af2-850f-42736b1afca0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:57.049359 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049299    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b984e1a2-de74-4af2-850f-42736b1afca0" (UID: "b984e1a2-de74-4af2-850f-42736b1afca0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:57.049359 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049333    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.049576 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.049545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b984e1a2-de74-4af2-850f-42736b1afca0" (UID: "b984e1a2-de74-4af2-850f-42736b1afca0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:57.051215 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.051195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw" (OuterVolumeSpecName: "kube-api-access-82hhw") pod "b984e1a2-de74-4af2-850f-42736b1afca0" (UID: "b984e1a2-de74-4af2-850f-42736b1afca0"). InnerVolumeSpecName "kube-api-access-82hhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:57.051325 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.051308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b984e1a2-de74-4af2-850f-42736b1afca0" (UID: "b984e1a2-de74-4af2-850f-42736b1afca0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:57.150225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.150190    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b984e1a2-de74-4af2-850f-42736b1afca0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.150225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.150215    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b984e1a2-de74-4af2-850f-42736b1afca0-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.150225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.150224    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hhw\" (UniqueName: \"kubernetes.io/projected/b984e1a2-de74-4af2-850f-42736b1afca0-kube-api-access-82hhw\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.150225 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.150233    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b984e1a2-de74-4af2-850f-42736b1afca0-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:56:57.743871 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.743842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" path="/var/lib/kubelet/pods/656f7793-475e-4a3f-8731-a8b7a64d58dc/volumes"
Sep 16 18:56:57.803341 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.803314    2580 generic.go:356] "Generic (PLEG): container finished" podID="b984e1a2-de74-4af2-850f-42736b1afca0" containerID="f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51" exitCode=0
Sep 16 18:56:57.803491 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.803362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerDied","Data":"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"}
Sep 16 18:56:57.803491 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.803376    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"
Sep 16 18:56:57.803491 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.803415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv" event={"ID":"b984e1a2-de74-4af2-850f-42736b1afca0","Type":"ContainerDied","Data":"ede2a2848137d8b9a45556eedb65a3f766dc091a8ab762e5e4b4a6135a8abe7b"}
Sep 16 18:56:57.803491 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.803440    2580 scope.go:117] "RemoveContainer" containerID="efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"
Sep 16 18:56:57.811856 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.811838    2580 scope.go:117] "RemoveContainer" containerID="f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"
Sep 16 18:56:57.818918 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.818902    2580 scope.go:117] "RemoveContainer" containerID="e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"
Sep 16 18:56:57.821783 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.821762    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:56:57.826244 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.826221    2580 scope.go:117] "RemoveContainer" containerID="efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"
Sep 16 18:56:57.826647 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:57.826628    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54\": container with ID starting with efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54 not found: ID does not exist" containerID="efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"
Sep 16 18:56:57.826732 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.826659    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54"} err="failed to get container status \"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54\": rpc error: code = NotFound desc = could not find container \"efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54\": container with ID starting with efd859577f4d0b11c79d7146a57850cd792e1c304f5fd14ade8142d8d9436a54 not found: ID does not exist"
Sep 16 18:56:57.826732 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.826681    2580 scope.go:117] "RemoveContainer" containerID="f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"
Sep 16 18:56:57.826944 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:57.826928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51\": container with ID starting with f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51 not found: ID does not exist" containerID="f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"
Sep 16 18:56:57.827035 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.826952    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51"} err="failed to get container status \"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51\": rpc error: code = NotFound desc = could not find container \"f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51\": container with ID starting with f1bba431a40f3d1c4879f0843a8b3998839a7a3ca66d5c94caadcb777c465c51 not found: ID does not exist"
Sep 16 18:56:57.827035 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.826972    2580 scope.go:117] "RemoveContainer" containerID="e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"
Sep 16 18:56:57.827249 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:56:57.827234    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c\": container with ID starting with e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c not found: ID does not exist" containerID="e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"
Sep 16 18:56:57.827331 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.827255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c"} err="failed to get container status \"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c\": rpc error: code = NotFound desc = could not find container \"e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c\": container with ID starting with e74658ceff56442a2ba5a3edf53d755c0403ed4d50b3f8d61e19003ee351d98c not found: ID does not exist"
Sep 16 18:56:57.828130 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:57.828114    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2e185-predictor-8596c88c8d-4fjhv"]
Sep 16 18:56:59.743768 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:56:59.743726    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" path="/var/lib/kubelet/pods/b984e1a2-de74-4af2-850f-42736b1afca0/volumes"
Sep 16 18:57:04.725436 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:04.725400    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:57:14.725123 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:14.725083    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:57:17.793145 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:17.793107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.724778 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:24.724721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:57:29.392366 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:29.392335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:34.724881 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:34.724841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:57:44.724598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:44.724552    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:57:53.744688 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:57:53.744619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:58:12.240356 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:12.240307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:12.256062 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:12.256027    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:58:12.256307 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:12.256283    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" containerID="cri-o://6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e" gracePeriod=30
Sep 16 18:58:12.256391 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:12.256351    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kube-rbac-proxy" containerID="cri-o://80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1" gracePeriod=30
Sep 16 18:58:12.342970 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:12.342938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:13.121698 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:13.121668    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerID="80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1" exitCode=2
Sep 16 18:58:13.121885 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:13.121736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerDied","Data":"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"}
Sep 16 18:58:13.740460 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:13.740419    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 16 18:58:14.720083 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:14.720044    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 16 18:58:16.907076 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.907053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:58:16.967140 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967052    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls\") pod \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") "
Sep 16 18:58:16.967140 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert\") pod \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") "
Sep 16 18:58:16.967352 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config\") pod \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") "
Sep 16 18:58:16.967352 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8ngd\" (UniqueName: \"kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd\") pod \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") "
Sep 16 18:58:16.967352 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967234    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location\") pod \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\" (UID: \"9ad02d13-005f-4a3d-86a5-8a26f3aac79e\") "
Sep 16 18:58:16.967592 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ad02d13-005f-4a3d-86a5-8a26f3aac79e" (UID: "9ad02d13-005f-4a3d-86a5-8a26f3aac79e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:16.967592 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967542    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ad02d13-005f-4a3d-86a5-8a26f3aac79e" (UID: "9ad02d13-005f-4a3d-86a5-8a26f3aac79e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:16.967848 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.967744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ad02d13-005f-4a3d-86a5-8a26f3aac79e" (UID: "9ad02d13-005f-4a3d-86a5-8a26f3aac79e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:16.969405 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.969365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ad02d13-005f-4a3d-86a5-8a26f3aac79e" (UID: "9ad02d13-005f-4a3d-86a5-8a26f3aac79e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:16.969689 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:16.969664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd" (OuterVolumeSpecName: "kube-api-access-r8ngd") pod "9ad02d13-005f-4a3d-86a5-8a26f3aac79e" (UID: "9ad02d13-005f-4a3d-86a5-8a26f3aac79e"). InnerVolumeSpecName "kube-api-access-r8ngd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:17.067795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.067764    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.067795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.067790    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.067795 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.067800    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.068045 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.067809    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ngd\" (UniqueName: \"kubernetes.io/projected/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kube-api-access-r8ngd\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.068045 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.067818    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ad02d13-005f-4a3d-86a5-8a26f3aac79e-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 18:58:17.139720 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.139683    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerID="6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e" exitCode=0
Sep 16 18:58:17.139889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.139725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerDied","Data":"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"}
Sep 16 18:58:17.139889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.139748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"
Sep 16 18:58:17.139889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.139798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9" event={"ID":"9ad02d13-005f-4a3d-86a5-8a26f3aac79e","Type":"ContainerDied","Data":"3e5ba0d114307329f7b9a26c11881b544e3b5778efc34e506cfc4cd0099440cd"}
Sep 16 18:58:17.139889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.139824    2580 scope.go:117] "RemoveContainer" containerID="80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"
Sep 16 18:58:17.149176 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.149157    2580 scope.go:117] "RemoveContainer" containerID="6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"
Sep 16 18:58:17.156648 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.156626    2580 scope.go:117] "RemoveContainer" containerID="69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"
Sep 16 18:58:17.162820 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.162798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:58:17.164291 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.164273    2580 scope.go:117] "RemoveContainer" containerID="80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"
Sep 16 18:58:17.164669 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:58:17.164642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1\": container with ID starting with 80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1 not found: ID does not exist" containerID="80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"
Sep 16 18:58:17.164772 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.164680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1"} err="failed to get container status \"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1\": rpc error: code = NotFound desc = could not find container \"80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1\": container with ID starting with 80cb2b68b807efd1cdb98d1f456c20222d3e3bf6da5dfad7e250c5874276f8f1 not found: ID does not exist"
Sep 16 18:58:17.164772 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.164707    2580 scope.go:117] "RemoveContainer" containerID="6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"
Sep 16 18:58:17.165232 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:58:17.165210    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e\": container with ID starting with 6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e not found: ID does not exist" containerID="6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"
Sep 16 18:58:17.165232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.165234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e"} err="failed to get container status \"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e\": rpc error: code = NotFound desc = could not find container \"6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e\": container with ID starting with 6e3d3cca1e84475e36e86ddf3185898fe8a5d3e4b3b14ae21a322d0185e1c64e not found: ID does not exist"
Sep 16 18:58:17.165232 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.165249    2580 scope.go:117] "RemoveContainer" containerID="69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"
Sep 16 18:58:17.165530 ip-10-0-128-253 kubenswrapper[2580]: E0916 18:58:17.165510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd\": container with ID starting with 69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd not found: ID does not exist" containerID="69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"
Sep 16 18:58:17.165589 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.165535    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd"} err="failed to get container status \"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd\": rpc error: code = NotFound desc = could not find container \"69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd\": container with ID starting with 69a72dad2cc3b4cb8ec1f4da35b162b07e16a958accb0a44909034e9bc16a8bd not found: ID does not exist"
Sep 16 18:58:17.166617 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.166596    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-4txf9"]
Sep 16 18:58:17.744499 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:17.744468    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" path="/var/lib/kubelet/pods/9ad02d13-005f-4a3d-86a5-8a26f3aac79e/volumes"
Sep 16 18:58:22.528972 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.528934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.549640 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.549608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 18:58:22.550080 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550066    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="storage-initializer"
Sep 16 18:58:22.550124 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550084    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="storage-initializer"
Sep 16 18:58:22.550124 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550096    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container"
Sep 16 18:58:22.550124 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550106    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container"
Sep 16 18:58:22.550124 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550117    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container"
Sep 16 18:58:22.550124 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550125    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="storage-initializer"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550145    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="storage-initializer"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550156    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550176    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550184    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550193    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550201    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550217    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550224    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550240    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="storage-initializer"
Sep 16 18:58:22.550270 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550248    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="storage-initializer"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kserve-container"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b984e1a2-de74-4af2-850f-42736b1afca0" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550342    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550352    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kserve-container"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="656f7793-475e-4a3f-8731-a8b7a64d58dc" containerName="kube-rbac-proxy"
Sep 16 18:58:22.550598 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.550369    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad02d13-005f-4a3d-86a5-8a26f3aac79e" containerName="kserve-container"
Sep 16 18:58:22.555712 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.555694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.559927 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.559893    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:22.560075 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.559936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:58:22.560075 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.559961    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1321b3-kube-rbac-proxy-sar-config\""
Sep 16 18:58:22.560344 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.560325    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 18:58:22.560466 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.560330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:58:22.560466 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.560344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1321b3-predictor-serving-cert\""
Sep 16 18:58:22.560979 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.560965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:58:22.569634 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.569614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 18:58:22.606963 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.606934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.606963 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.606967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.607196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.607000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.607196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.607086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rkg\" (UniqueName: \"kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.607196 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.607116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708501 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708501 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708707 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708707 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rkg\" (UniqueName: \"kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708707 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.708924 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.708899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.709252 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.709222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.709252 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.709226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.711103 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.711072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.722708 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.722687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rkg\" (UniqueName: \"kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg\") pod \"isvc-primary-1321b3-predictor-54575bd8b5-bm79l\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") " pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:22.752394 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.752365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.856070 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.856015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.866865 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:22.866843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:23.138651 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:23.138629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 18:58:23.141209 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:58:23.141181    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5574fe2b_3ace_441f_a93d_88e2fa1e7af7.slice/crio-8b27e33641aa06228719b196595c225dd8abe27c1df38f0f46f66c45a861bde5 WatchSource:0}: Error finding container 8b27e33641aa06228719b196595c225dd8abe27c1df38f0f46f66c45a861bde5: Status 404 returned error can't find the container with id 8b27e33641aa06228719b196595c225dd8abe27c1df38f0f46f66c45a861bde5
Sep 16 18:58:23.164185 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:23.164017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerStarted","Data":"8b27e33641aa06228719b196595c225dd8abe27c1df38f0f46f66c45a861bde5"}
Sep 16 18:58:24.173543 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:24.173506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerStarted","Data":"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"}
Sep 16 18:58:27.186563 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:27.186533    2580 generic.go:356] "Generic (PLEG): container finished" podID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerID="55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9" exitCode=0
Sep 16 18:58:27.186890 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:27.186609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerDied","Data":"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"}
Sep 16 18:58:28.192061 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:28.192027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerStarted","Data":"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"}
Sep 16 18:58:28.192426 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:28.192097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerStarted","Data":"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"}
Sep 16 18:58:28.192426 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:28.192286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:28.211758 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:28.211713    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podStartSLOduration=2.378279953 podStartE2EDuration="6.211698569s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833418616s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:28.210455561 +0000 UTC m=+838.955951357" watchObservedRunningTime="2026-09-16 18:58:28.211698569 +0000 UTC m=+838.957194366"
Sep 16 18:58:29.197059 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:29.197026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:29.198419 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:29.198393    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:58:30.201120 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:30.201073    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:58:32.415655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:32.415614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:32.731629 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:32.731551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.205009 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:35.204965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:58:35.205414 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:35.205390    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:58:42.528660 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.528623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:42.615889 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.615854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:42.690802 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.690773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:42.700883 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.700858    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 18:58:42.705767 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.705752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.708358 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.708339    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-16e43-predictor-serving-cert\""
Sep 16 18:58:42.708609 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.708593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-16e43-kube-rbac-proxy-sar-config\""
Sep 16 18:58:42.713457 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.713433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 18:58:42.755467 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.755443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.755562 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.755483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.755562 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.755508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfk8q\" (UniqueName: \"kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.785513 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.785455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:42.856768 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.856741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.856878 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.856777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.856878 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.856867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfk8q\" (UniqueName: \"kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.857433 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.857415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.859445 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.859428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.865773 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.865754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfk8q\" (UniqueName: \"kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q\") pod \"message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") " pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:42.891118 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:42.891095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:43.018854 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:43.018815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:43.288830 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:43.288693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 18:58:43.291301 ip-10-0-128-253 kubenswrapper[2580]: W0916 18:58:43.291265    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda863e1a6_047c_4a82_9a81_d036d56f34fb.slice/crio-7e446f484334b293febd28787472146afe83826820fd4f427b4216cf903c4c00 WatchSource:0}: Error finding container 7e446f484334b293febd28787472146afe83826820fd4f427b4216cf903c4c00: Status 404 returned error can't find the container with id 7e446f484334b293febd28787472146afe83826820fd4f427b4216cf903c4c00
Sep 16 18:58:44.257438 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:44.257404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerStarted","Data":"7e446f484334b293febd28787472146afe83826820fd4f427b4216cf903c4c00"}
Sep 16 18:58:45.206280 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:45.206242    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:58:45.263508 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:45.263475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerStarted","Data":"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"}
Sep 16 18:58:45.263876 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:45.263541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerStarted","Data":"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"}
Sep 16 18:58:45.263876 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:45.263594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:45.281624 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:45.281582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" podStartSLOduration=2.040060908 podStartE2EDuration="3.281569229s" totalImagesPullingTime="1.241508321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:45.279811997 +0000 UTC m=+856.025307805" watchObservedRunningTime="2026-09-16 18:58:45.281569229 +0000 UTC m=+856.027065026"
Sep 16 18:58:46.267683 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:46.267654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:46.269287 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:46.269265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:53.279480 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:53.279453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 18:58:55.205734 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:58:55.205690    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:02.680655 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:02.680620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:02.797860 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:02.797826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:05.205459 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:05.205409    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:15.205537 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:15.205502    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:25.206385 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:25.206296    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:35.205891 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:35.205851    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:35.740366 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:35.740329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 16 18:59:45.745037 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:45.745010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 18:59:50.643007 ip-10-0-128-253 kubenswrapper[2580]: I0916 18:59:50.642958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:07.646267 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.646232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:07.670042 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.670012    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:07.673664 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.673649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.676384 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.676363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1321b3-kube-rbac-proxy-sar-config\""
Sep 16 19:00:07.676497 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.676460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1321b3-predictor-serving-cert\""
Sep 16 19:00:07.676819 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.676800    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-1321b3-dockercfg-tvpfq\""
Sep 16 19:00:07.676923 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.676807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-1321b3\""
Sep 16 19:00:07.682580 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.682552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:07.708530 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.708498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.708530 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.708528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.714540 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.708570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.714540 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.709068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.714540 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.709184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsk99\" (UniqueName: \"kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810238 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810238 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810472 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsk99\" (UniqueName: \"kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810540 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810540 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810661 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:07.810641    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-serving-cert: secret "isvc-secondary-1321b3-predictor-serving-cert" not found
Sep 16 19:00:07.810726 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.810664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.810726 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:07.810721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls podName:e1fb7994-6365-4c00-9ab6-c3dd6d9dd066 nodeName:}" failed. No retries permitted until 2026-09-16 19:00:08.310700463 +0000 UTC m=+939.056196251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls") pod "isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066") : secret "isvc-secondary-1321b3-predictor-serving-cert" not found
Sep 16 19:00:07.811047 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.811024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.811149 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.811133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:07.819471 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:07.819448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsk99\" (UniqueName: \"kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:08.315527 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:08.315490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:08.319145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:08.319094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") pod \"isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") " pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:08.584163 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:08.584082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:08.860231 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:08.860207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:08.862233 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:00:08.862202    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fb7994_6365_4c00_9ab6_c3dd6d9dd066.slice/crio-4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b WatchSource:0}: Error finding container 4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b: Status 404 returned error can't find the container with id 4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b
Sep 16 19:00:09.588320 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:09.588276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerStarted","Data":"9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"}
Sep 16 19:00:09.588466 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:09.588349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerStarted","Data":"4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b"}
Sep 16 19:00:10.779699 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:10.779663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:15.615177 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:15.615149    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/0.log"
Sep 16 19:00:15.615575 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:15.615187    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerID="9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5" exitCode=1
Sep 16 19:00:15.615575 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:15.615266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerDied","Data":"9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"}
Sep 16 19:00:16.621102 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:16.621075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/0.log"
Sep 16 19:00:16.621476 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:16.621149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerStarted","Data":"f702ba9bc92e8084ab810d73d20b1a166eaee949ab8eed3728947ac6e4a08412"}
Sep 16 19:00:20.638678 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.638648    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/1.log"
Sep 16 19:00:20.639112 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.639085    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/0.log"
Sep 16 19:00:20.639179 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.639124    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerID="f702ba9bc92e8084ab810d73d20b1a166eaee949ab8eed3728947ac6e4a08412" exitCode=1
Sep 16 19:00:20.639222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.639196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerDied","Data":"f702ba9bc92e8084ab810d73d20b1a166eaee949ab8eed3728947ac6e4a08412"}
Sep 16 19:00:20.639314 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.639302    2580 scope.go:117] "RemoveContainer" containerID="9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"
Sep 16 19:00:20.639735 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.639718    2580 scope.go:117] "RemoveContainer" containerID="9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"
Sep 16 19:00:20.650034 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:20.650007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_kserve-ci-e2e-test(e1fb7994-6365-4c00-9ab6-c3dd6d9dd066)\"" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"
Sep 16 19:00:20.650271 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:20.650249    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_kserve-ci-e2e-test_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066_0 in pod sandbox 4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b from index: no such id: '9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5'" containerID="9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"
Sep 16 19:00:20.650332 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:20.650283    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_kserve-ci-e2e-test_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066_0 in pod sandbox 4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b from index: no such id: '9de05060aabb964c50065012936a5701c21b073363c22ac4ee13764a48b9ecd5'"
Sep 16 19:00:21.645004 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:21.644963    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/1.log"
Sep 16 19:00:27.723596 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.723562    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:27.738865 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.738830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:27.753813 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.753785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 19:00:27.754386 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.754355    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container" containerID="cri-o://8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e" gracePeriod=30
Sep 16 19:00:27.754708 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.754684    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kube-rbac-proxy" containerID="cri-o://e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3" gracePeriod=30
Sep 16 19:00:27.837481 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.837452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:27.875376 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.875350    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/1.log"
Sep 16 19:00:27.877390 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.877375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:27.972248 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location\") pod \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") "
Sep 16 19:00:27.972248 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsk99\" (UniqueName: \"kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99\") pod \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") "
Sep 16 19:00:27.972406 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config\") pod \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") "
Sep 16 19:00:27.972406 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert\") pod \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") "
Sep 16 19:00:27.972406 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") pod \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\" (UID: \"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066\") "
Sep 16 19:00:27.972531 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:27.972603 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972586    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:27.972658 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:27.972706 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.972644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:27.974402 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.974354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99" (OuterVolumeSpecName: "kube-api-access-hsk99") pod "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"). InnerVolumeSpecName "kube-api-access-hsk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:27.974567 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:27.974550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" (UID: "e1fb7994-6365-4c00-9ab6-c3dd6d9dd066"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:28.073443 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.073418    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsk99\" (UniqueName: \"kubernetes.io/projected/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-api-access-hsk99\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.073443 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.073439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.073590 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.073449    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.073590 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.073459    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.674744 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.674712    2580 generic.go:356] "Generic (PLEG): container finished" podID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerID="e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3" exitCode=2
Sep 16 19:00:28.674923 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.674787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerDied","Data":"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"}
Sep 16 19:00:28.675881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.675862    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f_e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/storage-initializer/1.log"
Sep 16 19:00:28.676007 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.675945    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"
Sep 16 19:00:28.676007 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.675953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f" event={"ID":"e1fb7994-6365-4c00-9ab6-c3dd6d9dd066","Type":"ContainerDied","Data":"4b6b0554ed0061657a0fe279c15b3d424402da5f1dcc03ae01d226c018f4ea2b"}
Sep 16 19:00:28.676085 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.676014    2580 scope.go:117] "RemoveContainer" containerID="f702ba9bc92e8084ab810d73d20b1a166eaee949ab8eed3728947ac6e4a08412"
Sep 16 19:00:28.712161 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.712136    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:28.715725 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:28.715705    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1321b3-predictor-79cf9f586b-5sw8f"]
Sep 16 19:00:29.744131 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:29.744104    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" path="/var/lib/kubelet/pods/e1fb7994-6365-4c00-9ab6-c3dd6d9dd066/volumes"
Sep 16 19:00:30.202098 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:30.202053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 16 19:00:32.195918 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.195889    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 19:00:32.305023 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.304942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config\") pod \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") "
Sep 16 19:00:32.305023 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert\") pod \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") "
Sep 16 19:00:32.305170 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rkg\" (UniqueName: \"kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg\") pod \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") "
Sep 16 19:00:32.305170 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location\") pod \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") "
Sep 16 19:00:32.305170 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls\") pod \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\" (UID: \"5574fe2b-3ace-441f-a93d-88e2fa1e7af7\") "
Sep 16 19:00:32.305447 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5574fe2b-3ace-441f-a93d-88e2fa1e7af7" (UID: "5574fe2b-3ace-441f-a93d-88e2fa1e7af7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:32.305509 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5574fe2b-3ace-441f-a93d-88e2fa1e7af7" (UID: "5574fe2b-3ace-441f-a93d-88e2fa1e7af7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:32.305509 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.305455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5574fe2b-3ace-441f-a93d-88e2fa1e7af7" (UID: "5574fe2b-3ace-441f-a93d-88e2fa1e7af7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:32.307310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.307278    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg" (OuterVolumeSpecName: "kube-api-access-52rkg") pod "5574fe2b-3ace-441f-a93d-88e2fa1e7af7" (UID: "5574fe2b-3ace-441f-a93d-88e2fa1e7af7"). InnerVolumeSpecName "kube-api-access-52rkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:32.307310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.307287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5574fe2b-3ace-441f-a93d-88e2fa1e7af7" (UID: "5574fe2b-3ace-441f-a93d-88e2fa1e7af7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:32.405618 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.405593    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.405618 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.405613    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.405747 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.405623    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rkg\" (UniqueName: \"kubernetes.io/projected/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kube-api-access-52rkg\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.405747 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.405632    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.405747 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.405641    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5574fe2b-3ace-441f-a93d-88e2fa1e7af7-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:32.694885 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.694848    2580 generic.go:356] "Generic (PLEG): container finished" podID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerID="8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e" exitCode=0
Sep 16 19:00:32.695099 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.694960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"
Sep 16 19:00:32.695099 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.695009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerDied","Data":"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"}
Sep 16 19:00:32.695218 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.695117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l" event={"ID":"5574fe2b-3ace-441f-a93d-88e2fa1e7af7","Type":"ContainerDied","Data":"8b27e33641aa06228719b196595c225dd8abe27c1df38f0f46f66c45a861bde5"}
Sep 16 19:00:32.695218 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.695160    2580 scope.go:117] "RemoveContainer" containerID="e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"
Sep 16 19:00:32.704192 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.704175    2580 scope.go:117] "RemoveContainer" containerID="8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"
Sep 16 19:00:32.711656 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.711645    2580 scope.go:117] "RemoveContainer" containerID="55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"
Sep 16 19:00:32.718749 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.718732    2580 scope.go:117] "RemoveContainer" containerID="e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"
Sep 16 19:00:32.719109 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:32.719062    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3\": container with ID starting with e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3 not found: ID does not exist" containerID="e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"
Sep 16 19:00:32.719241 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.719109    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3"} err="failed to get container status \"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3\": rpc error: code = NotFound desc = could not find container \"e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3\": container with ID starting with e5d1e5f45a42f26b1496bbf99bd4db37e4dc493bfccae32bbcf18b824bd7ccd3 not found: ID does not exist"
Sep 16 19:00:32.719241 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.719135    2580 scope.go:117] "RemoveContainer" containerID="8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"
Sep 16 19:00:32.719513 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:32.719480    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e\": container with ID starting with 8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e not found: ID does not exist" containerID="8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"
Sep 16 19:00:32.719641 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.719618    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e"} err="failed to get container status \"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e\": rpc error: code = NotFound desc = could not find container \"8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e\": container with ID starting with 8f5b5e82685ee726e0f81cee8c0a852b7214eac1663c5e8af0ece8879ebec99e not found: ID does not exist"
Sep 16 19:00:32.719717 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.719644    2580 scope.go:117] "RemoveContainer" containerID="55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"
Sep 16 19:00:32.719945 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:32.719929    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9\": container with ID starting with 55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9 not found: ID does not exist" containerID="55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"
Sep 16 19:00:32.720050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.719951    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9"} err="failed to get container status \"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9\": rpc error: code = NotFound desc = could not find container \"55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9\": container with ID starting with 55556977f4b9d9f208c23fcf29afc5bcc5c82e5ddba6904f1c367700ca4719b9 not found: ID does not exist"
Sep 16 19:00:32.720938 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.720920    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 19:00:32.724782 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:32.724759    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1321b3-predictor-54575bd8b5-bm79l"]
Sep 16 19:00:33.004186 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.004100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:33.025715 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.025688    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:33.026009 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.025996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.026050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026048    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026053    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kube-rbac-proxy"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026069    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kube-rbac-proxy"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026114    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026124    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kserve-container"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026132    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" containerName="kube-rbac-proxy"
Sep 16 19:00:33.026222 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.026221    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fb7994-6365-4c00-9ab6-c3dd6d9dd066" containerName="storage-initializer"
Sep 16 19:00:33.030703 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.030686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.033558 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.033532    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:33.033558 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.033545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-86458d-dockercfg-2ltf7\""
Sep 16 19:00:33.033558 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.033555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-86458d-predictor-serving-cert\""
Sep 16 19:00:33.033773 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.033545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-86458d\""
Sep 16 19:00:33.033773 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.033572    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-86458d-kube-rbac-proxy-sar-config\""
Sep 16 19:00:33.044702 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.044679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:33.110763 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.110733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.110929 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.110773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.110929 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.110799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.110929 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.110822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.110929 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.110907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvfqs\" (UniqueName: \"kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212200 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvfqs\" (UniqueName: \"kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:33.212351    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-serving-cert: secret "isvc-init-fail-86458d-predictor-serving-cert" not found
Sep 16 19:00:33.212623 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:33.212414    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls podName:57b78258-29f8-4bd5-8ec3-00ba51b5284b nodeName:}" failed. No retries permitted until 2026-09-16 19:00:33.712395384 +0000 UTC m=+964.457891172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls") pod "isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b") : secret "isvc-init-fail-86458d-predictor-serving-cert" not found
Sep 16 19:00:33.212893 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.212757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.213034 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.213015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.213071 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.213015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.221627 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.221605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvfqs\" (UniqueName: \"kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.299937 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.299868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:33.715373 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.715347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.717794 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.717773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") pod \"isvc-init-fail-86458d-predictor-68b9d45998-bk9q6\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") " pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:33.744221 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.744197    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5574fe2b-3ace-441f-a93d-88e2fa1e7af7" path="/var/lib/kubelet/pods/5574fe2b-3ace-441f-a93d-88e2fa1e7af7/volumes"
Sep 16 19:00:33.941005 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:33.940959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:34.213317 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:34.213289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:34.216495 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:00:34.216468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b78258_29f8_4bd5_8ec3_00ba51b5284b.slice/crio-58b845b20355eef259a866ecc5b4c846098645566473a49021c50447a57f51c9 WatchSource:0}: Error finding container 58b845b20355eef259a866ecc5b4c846098645566473a49021c50447a57f51c9: Status 404 returned error can't find the container with id 58b845b20355eef259a866ecc5b4c846098645566473a49021c50447a57f51c9
Sep 16 19:00:34.706589 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:34.706553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerStarted","Data":"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"}
Sep 16 19:00:34.706743 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:34.706612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerStarted","Data":"58b845b20355eef259a866ecc5b4c846098645566473a49021c50447a57f51c9"}
Sep 16 19:00:37.721270 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.721244    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/0.log"
Sep 16 19:00:37.721623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.721279    2580 generic.go:356] "Generic (PLEG): container finished" podID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerID="0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef" exitCode=1
Sep 16 19:00:37.721623 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.721336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerDied","Data":"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"}
Sep 16 19:00:37.725939 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.725906    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg_a863e1a6-047c-4a82-9a81-d036d56f34fb/kserve-container/0.log"
Sep 16 19:00:37.928185 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.928152    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 19:00:37.928475 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.928450    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kserve-container" containerID="cri-o://0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812" gracePeriod=30
Sep 16 19:00:37.928559 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.928533    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kube-rbac-proxy" containerID="cri-o://295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8" gracePeriod=30
Sep 16 19:00:37.973200 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:37.973171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:38.075897 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.075862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:38.176386 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.176360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:38.178106 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.178088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 19:00:38.250041 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.249952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:38.349135 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.349103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config\") pod \"a863e1a6-047c-4a82-9a81-d036d56f34fb\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") "
Sep 16 19:00:38.349279 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.349146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfk8q\" (UniqueName: \"kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q\") pod \"a863e1a6-047c-4a82-9a81-d036d56f34fb\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") "
Sep 16 19:00:38.349279 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.349217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls\") pod \"a863e1a6-047c-4a82-9a81-d036d56f34fb\" (UID: \"a863e1a6-047c-4a82-9a81-d036d56f34fb\") "
Sep 16 19:00:38.349445 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.349423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a863e1a6-047c-4a82-9a81-d036d56f34fb" (UID: "a863e1a6-047c-4a82-9a81-d036d56f34fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:38.351338 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.351316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q" (OuterVolumeSpecName: "kube-api-access-wfk8q") pod "a863e1a6-047c-4a82-9a81-d036d56f34fb" (UID: "a863e1a6-047c-4a82-9a81-d036d56f34fb"). InnerVolumeSpecName "kube-api-access-wfk8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:38.351423 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.351332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a863e1a6-047c-4a82-9a81-d036d56f34fb" (UID: "a863e1a6-047c-4a82-9a81-d036d56f34fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:38.449760 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.449730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfk8q\" (UniqueName: \"kubernetes.io/projected/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-api-access-wfk8q\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.449760 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.449756    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a863e1a6-047c-4a82-9a81-d036d56f34fb-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.449760 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.449765    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a863e1a6-047c-4a82-9a81-d036d56f34fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.726596 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.726573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/0.log"
Sep 16 19:00:38.727021 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.726699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerStarted","Data":"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"}
Sep 16 19:00:38.728259 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728235    2580 generic.go:356] "Generic (PLEG): container finished" podID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerID="295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8" exitCode=2
Sep 16 19:00:38.728259 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728254    2580 generic.go:356] "Generic (PLEG): container finished" podID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerID="0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812" exitCode=2
Sep 16 19:00:38.728436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerDied","Data":"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"}
Sep 16 19:00:38.728436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728277    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"
Sep 16 19:00:38.728436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerDied","Data":"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"}
Sep 16 19:00:38.728436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg" event={"ID":"a863e1a6-047c-4a82-9a81-d036d56f34fb","Type":"ContainerDied","Data":"7e446f484334b293febd28787472146afe83826820fd4f427b4216cf903c4c00"}
Sep 16 19:00:38.728436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.728353    2580 scope.go:117] "RemoveContainer" containerID="295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"
Sep 16 19:00:38.737756 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.737687    2580 scope.go:117] "RemoveContainer" containerID="0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"
Sep 16 19:00:38.746225 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.746204    2580 scope.go:117] "RemoveContainer" containerID="295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"
Sep 16 19:00:38.746606 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:38.746584    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8\": container with ID starting with 295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8 not found: ID does not exist" containerID="295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"
Sep 16 19:00:38.746705 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.746617    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"} err="failed to get container status \"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8\": rpc error: code = NotFound desc = could not find container \"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8\": container with ID starting with 295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8 not found: ID does not exist"
Sep 16 19:00:38.746705 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.746642    2580 scope.go:117] "RemoveContainer" containerID="0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"
Sep 16 19:00:38.746954 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:38.746934    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812\": container with ID starting with 0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812 not found: ID does not exist" containerID="0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"
Sep 16 19:00:38.747050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.746964    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"} err="failed to get container status \"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812\": rpc error: code = NotFound desc = could not find container \"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812\": container with ID starting with 0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812 not found: ID does not exist"
Sep 16 19:00:38.747050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.746999    2580 scope.go:117] "RemoveContainer" containerID="295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"
Sep 16 19:00:38.747266 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.747245    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8"} err="failed to get container status \"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8\": rpc error: code = NotFound desc = could not find container \"295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8\": container with ID starting with 295a9847e595d586eca37d46c4af10b2aaf569584d298ba6eb6180c6d3dcdbf8 not found: ID does not exist"
Sep 16 19:00:38.747335 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.747267    2580 scope.go:117] "RemoveContainer" containerID="0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"
Sep 16 19:00:38.747552 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.747533    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812"} err="failed to get container status \"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812\": rpc error: code = NotFound desc = could not find container \"0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812\": container with ID starting with 0db01a493c6631d6436b5d4b3b988d8a1faa80286708aa05e92cefea735fb812 not found: ID does not exist"
Sep 16 19:00:38.762119 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.762100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 19:00:38.766055 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:38.766033    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-16e43-predictor-7fc75fbc47-t5jzg"]
Sep 16 19:00:39.744356 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:39.744326    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" path="/var/lib/kubelet/pods/a863e1a6-047c-4a82-9a81-d036d56f34fb/volumes"
Sep 16 19:00:42.947399 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:42.947358    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:42.947857 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:42.947601    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer" containerID="cri-o://21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd" gracePeriod=30
Sep 16 19:00:42.974448 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:42.974424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:43.896130 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.896108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/1.log"
Sep 16 19:00:43.896507 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.896491    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/0.log"
Sep 16 19:00:43.898865 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.898848    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:43.987431 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config\") pod \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") "
Sep 16 19:00:43.987431 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location\") pod \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") "
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvfqs\" (UniqueName: \"kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs\") pod \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") "
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert\") pod \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") "
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987505    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") pod \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\" (UID: \"57b78258-29f8-4bd5-8ec3-00ba51b5284b\") "
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57b78258-29f8-4bd5-8ec3-00ba51b5284b" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57b78258-29f8-4bd5-8ec3-00ba51b5284b" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:43.987838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.987817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57b78258-29f8-4bd5-8ec3-00ba51b5284b" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:43.989568 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.989549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs" (OuterVolumeSpecName: "kube-api-access-cvfqs") pod "57b78258-29f8-4bd5-8ec3-00ba51b5284b" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b"). InnerVolumeSpecName "kube-api-access-cvfqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:43.989696 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:43.989679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57b78258-29f8-4bd5-8ec3-00ba51b5284b" (UID: "57b78258-29f8-4bd5-8ec3-00ba51b5284b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:44.088915 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.088888    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.088915 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.088912    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.089065 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.088928    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvfqs\" (UniqueName: \"kubernetes.io/projected/57b78258-29f8-4bd5-8ec3-00ba51b5284b-kube-api-access-cvfqs\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.089065 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.088941    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b78258-29f8-4bd5-8ec3-00ba51b5284b-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.089065 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.088956    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b78258-29f8-4bd5-8ec3-00ba51b5284b-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.757513 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.757488    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/1.log"
Sep 16 19:00:44.757890 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.757875    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-86458d-predictor-68b9d45998-bk9q6_57b78258-29f8-4bd5-8ec3-00ba51b5284b/storage-initializer/0.log"
Sep 16 19:00:44.757960 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.757910    2580 generic.go:356] "Generic (PLEG): container finished" podID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerID="21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd" exitCode=1
Sep 16 19:00:44.758034 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.757976    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"
Sep 16 19:00:44.758087 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.758022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerDied","Data":"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"}
Sep 16 19:00:44.758087 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.758083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6" event={"ID":"57b78258-29f8-4bd5-8ec3-00ba51b5284b","Type":"ContainerDied","Data":"58b845b20355eef259a866ecc5b4c846098645566473a49021c50447a57f51c9"}
Sep 16 19:00:44.758200 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.758112    2580 scope.go:117] "RemoveContainer" containerID="21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"
Sep 16 19:00:44.766564 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.766544    2580 scope.go:117] "RemoveContainer" containerID="0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"
Sep 16 19:00:44.774060 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.774045    2580 scope.go:117] "RemoveContainer" containerID="21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"
Sep 16 19:00:44.774319 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:44.774295    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd\": container with ID starting with 21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd not found: ID does not exist" containerID="21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"
Sep 16 19:00:44.774367 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.774320    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd"} err="failed to get container status \"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd\": rpc error: code = NotFound desc = could not find container \"21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd\": container with ID starting with 21c0aeb09d8be61e11dcc7072f01e36469d6333353ddc71475c68b870dc043cd not found: ID does not exist"
Sep 16 19:00:44.774367 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.774335    2580 scope.go:117] "RemoveContainer" containerID="0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"
Sep 16 19:00:44.774577 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:44.774561    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef\": container with ID starting with 0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef not found: ID does not exist" containerID="0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"
Sep 16 19:00:44.774609 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.774584    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef"} err="failed to get container status \"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef\": rpc error: code = NotFound desc = could not find container \"0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef\": container with ID starting with 0a195d2236f187e8b0acdaba46463b40070639afae2e45416d8d017396ce0fef not found: ID does not exist"
Sep 16 19:00:44.798818 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.798796    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:44.802398 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:44.802376    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-86458d-predictor-68b9d45998-bk9q6"]
Sep 16 19:00:45.744000 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:45.743954    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" path="/var/lib/kubelet/pods/57b78258-29f8-4bd5-8ec3-00ba51b5284b/volumes"
Sep 16 19:00:48.173778 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.173745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:48.210603 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.210567    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:00:48.211029 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211014    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kserve-container"
Sep 16 19:00:48.211076 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kserve-container"
Sep 16 19:00:48.211076 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211061    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211105    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kube-rbac-proxy"
Sep 16 19:00:48.211145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kube-rbac-proxy"
Sep 16 19:00:48.211145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211123    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211145 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211290 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211290 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b78258-29f8-4bd5-8ec3-00ba51b5284b" containerName="storage-initializer"
Sep 16 19:00:48.211290 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211227    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kserve-container"
Sep 16 19:00:48.211290 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.211238    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a863e1a6-047c-4a82-9a81-d036d56f34fb" containerName="kube-rbac-proxy"
Sep 16 19:00:48.215976 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.215957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.220686 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.220644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-aef9f-predictor-serving-cert\""
Sep 16 19:00:48.220807 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.220719    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:00:48.220807 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.220785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:00:48.221835 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.221816    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-aef9f-kube-rbac-proxy-sar-config\""
Sep 16 19:00:48.221918 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.221877    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:48.221918 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.221878    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:00:48.221918 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.221885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:00:48.229763 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.229742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:00:48.318162 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.318134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.318260 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.318185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.318260 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.318244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.318373 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.318287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv8l\" (UniqueName: \"kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.318373 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.318338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419279 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv8l\" (UniqueName: \"kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419416 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419416 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419416 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419416 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.419637 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:48.419492    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-serving-cert: secret "raw-sklearn-aef9f-predictor-serving-cert" not found
Sep 16 19:00:48.419637 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:00:48.419574    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls podName:79932bbe-63e6-4d9b-aeaf-023419f8639e nodeName:}" failed. No retries permitted until 2026-09-16 19:00:48.919555671 +0000 UTC m=+979.665051446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls") pod "raw-sklearn-aef9f-predictor-694d557f6d-dhznd" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e") : secret "raw-sklearn-aef9f-predictor-serving-cert" not found
Sep 16 19:00:48.419842 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.419820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.420050 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.420029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.420103 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.420062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.431618 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.431562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv8l\" (UniqueName: \"kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.741524 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.741440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:48.925271 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.925236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:48.927834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:48.927807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") pod \"raw-sklearn-aef9f-predictor-694d557f6d-dhznd\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") " pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:49.126133 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:49.126099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:49.399232 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:49.399211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:00:49.401248 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:00:49.401219    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79932bbe_63e6_4d9b_aeaf_023419f8639e.slice/crio-758eea3f61edab6300af4ca637e751779e18c2187e4dbcb927508eb423a585ec WatchSource:0}: Error finding container 758eea3f61edab6300af4ca637e751779e18c2187e4dbcb927508eb423a585ec: Status 404 returned error can't find the container with id 758eea3f61edab6300af4ca637e751779e18c2187e4dbcb927508eb423a585ec
Sep 16 19:00:49.778856 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:49.778764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerStarted","Data":"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"}
Sep 16 19:00:49.778856 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:49.778823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerStarted","Data":"758eea3f61edab6300af4ca637e751779e18c2187e4dbcb927508eb423a585ec"}
Sep 16 19:00:53.797844 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:53.797810    2580 generic.go:356] "Generic (PLEG): container finished" podID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerID="af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3" exitCode=0
Sep 16 19:00:53.798217 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:53.797867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerDied","Data":"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"}
Sep 16 19:00:54.804365 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.804330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerStarted","Data":"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"}
Sep 16 19:00:54.804762 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.804415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerStarted","Data":"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"}
Sep 16 19:00:54.804762 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.804729    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:54.804868 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.804789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:00:54.806056 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.806031    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:00:54.824255 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:54.824217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podStartSLOduration=3.234032488 podStartE2EDuration="6.82420541s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.590172922s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:54.823018893 +0000 UTC m=+985.568514689" watchObservedRunningTime="2026-09-16 19:00:54.82420541 +0000 UTC m=+985.569701207"
Sep 16 19:00:55.808672 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:00:55.808633    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:00.812807 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:00.812780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:01:00.813280 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:00.813253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:10.813838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:10.813798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:20.813405 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:20.813366    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:30.813923 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:30.813884    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:40.813902 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:40.813862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:01:50.813328 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:01:50.813283    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:02:00.813245 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:00.813207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:02:10.814094 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:10.814061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:02:18.331712 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:18.331679    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:02:18.332262 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:18.331921    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" containerID="cri-o://11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da" gracePeriod=30
Sep 16 19:02:18.332262 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:18.331947    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kube-rbac-proxy" containerID="cri-o://0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c" gracePeriod=30
Sep 16 19:02:18.354646 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:18.354621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:19.145335 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:19.145300    2580 generic.go:356] "Generic (PLEG): container finished" podID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerID="0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c" exitCode=2
Sep 16 19:02:19.145502 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:19.145371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerDied","Data":"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"}
Sep 16 19:02:20.809618 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:20.809523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 19:02:20.813964 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:20.813933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 19:02:22.676421 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.676400    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:02:22.737088 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert\") pod \"79932bbe-63e6-4d9b-aeaf-023419f8639e\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") "
Sep 16 19:02:22.737227 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") pod \"79932bbe-63e6-4d9b-aeaf-023419f8639e\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") "
Sep 16 19:02:22.737227 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwv8l\" (UniqueName: \"kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l\") pod \"79932bbe-63e6-4d9b-aeaf-023419f8639e\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") "
Sep 16 19:02:22.737227 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location\") pod \"79932bbe-63e6-4d9b-aeaf-023419f8639e\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") "
Sep 16 19:02:22.737393 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config\") pod \"79932bbe-63e6-4d9b-aeaf-023419f8639e\" (UID: \"79932bbe-63e6-4d9b-aeaf-023419f8639e\") "
Sep 16 19:02:22.737450 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79932bbe-63e6-4d9b-aeaf-023419f8639e" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:22.737493 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79932bbe-63e6-4d9b-aeaf-023419f8639e" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:22.737644 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737631    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79932bbe-63e6-4d9b-aeaf-023419f8639e-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:02:22.737689 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737647    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:02:22.737689 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.737646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79932bbe-63e6-4d9b-aeaf-023419f8639e" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:22.739898 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.739879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79932bbe-63e6-4d9b-aeaf-023419f8639e" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:22.739898 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.739889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l" (OuterVolumeSpecName: "kube-api-access-wwv8l") pod "79932bbe-63e6-4d9b-aeaf-023419f8639e" (UID: "79932bbe-63e6-4d9b-aeaf-023419f8639e"). InnerVolumeSpecName "kube-api-access-wwv8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:22.838457 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.838382    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:02:22.838457 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.838408    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79932bbe-63e6-4d9b-aeaf-023419f8639e-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:02:22.838457 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:22.838421    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv8l\" (UniqueName: \"kubernetes.io/projected/79932bbe-63e6-4d9b-aeaf-023419f8639e-kube-api-access-wwv8l\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:02:23.162895 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.162865    2580 generic.go:356] "Generic (PLEG): container finished" podID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerID="11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da" exitCode=0
Sep 16 19:02:23.163058 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.162920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"
Sep 16 19:02:23.163058 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.162947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerDied","Data":"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"}
Sep 16 19:02:23.163058 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.163036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd" event={"ID":"79932bbe-63e6-4d9b-aeaf-023419f8639e","Type":"ContainerDied","Data":"758eea3f61edab6300af4ca637e751779e18c2187e4dbcb927508eb423a585ec"}
Sep 16 19:02:23.163225 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.163065    2580 scope.go:117] "RemoveContainer" containerID="0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"
Sep 16 19:02:23.171955 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.171940    2580 scope.go:117] "RemoveContainer" containerID="11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"
Sep 16 19:02:23.179181 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.179163    2580 scope.go:117] "RemoveContainer" containerID="af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"
Sep 16 19:02:23.184219 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.184199    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:02:23.186595 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.186579    2580 scope.go:117] "RemoveContainer" containerID="0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"
Sep 16 19:02:23.186870 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:02:23.186853    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c\": container with ID starting with 0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c not found: ID does not exist" containerID="0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"
Sep 16 19:02:23.186921 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.186880    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c"} err="failed to get container status \"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c\": rpc error: code = NotFound desc = could not find container \"0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c\": container with ID starting with 0d81b4aa0f12f6e360d97493c63929ea642322c58f611997dbc8cdbf82c7649c not found: ID does not exist"
Sep 16 19:02:23.186921 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.186898    2580 scope.go:117] "RemoveContainer" containerID="11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"
Sep 16 19:02:23.187193 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:02:23.187175    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da\": container with ID starting with 11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da not found: ID does not exist" containerID="11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"
Sep 16 19:02:23.187243 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.187200    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da"} err="failed to get container status \"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da\": rpc error: code = NotFound desc = could not find container \"11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da\": container with ID starting with 11598f4e7f3ae3948368238a69d67f9873243de7340e1c0ef902118d66e215da not found: ID does not exist"
Sep 16 19:02:23.187243 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.187215    2580 scope.go:117] "RemoveContainer" containerID="af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"
Sep 16 19:02:23.187465 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:02:23.187441    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3\": container with ID starting with af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3 not found: ID does not exist" containerID="af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"
Sep 16 19:02:23.187529 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.187471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3"} err="failed to get container status \"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3\": rpc error: code = NotFound desc = could not find container \"af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3\": container with ID starting with af6768b1098db4eb34a5eccc8d52342e3d250d705c0b1608e9653c402ef704f3 not found: ID does not exist"
Sep 16 19:02:23.189722 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.189705    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-aef9f-predictor-694d557f6d-dhznd"]
Sep 16 19:02:23.424138 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.424066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:23.483812 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.483782    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:02:23.484120 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484107    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container"
Sep 16 19:02:23.484175 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484121    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container"
Sep 16 19:02:23.484175 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484134    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="storage-initializer"
Sep 16 19:02:23.484175 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484139    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="storage-initializer"
Sep 16 19:02:23.484175 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484149    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kube-rbac-proxy"
Sep 16 19:02:23.484175 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484154    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kube-rbac-proxy"
Sep 16 19:02:23.484328 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kube-rbac-proxy"
Sep 16 19:02:23.484328 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.484214    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" containerName="kserve-container"
Sep 16 19:02:23.488769 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.488752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.492660 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.492641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-20b49-predictor-serving-cert\""
Sep 16 19:02:23.492769 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.492694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:02:23.493465 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.493445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-20b49-kube-rbac-proxy-sar-config\""
Sep 16 19:02:23.493567 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.493475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:02:23.493567 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.493560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:02:23.493698 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.493646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:02:23.493698 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.493652    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:02:23.501217 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.501198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:02:23.540318 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.540292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:23.544263 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.544246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr7m\" (UniqueName: \"kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.544326 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.544278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.544375 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.544362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.544414 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.544384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.544449 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.544410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.644772 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.644747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr7m\" (UniqueName: \"kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.644868 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.644780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.644868 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.644843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.644958 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.644868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.644958 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.644897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.645344 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.645322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.645557 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.645540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.645630 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.645603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.647421 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.647389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.654356 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.654334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr7m\" (UniqueName: \"kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m\") pod \"raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:23.744424 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.744365    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79932bbe-63e6-4d9b-aeaf-023419f8639e" path="/var/lib/kubelet/pods/79932bbe-63e6-4d9b-aeaf-023419f8639e/volumes"
Sep 16 19:02:23.798689 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:23.798668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:24.072852 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:24.072832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:02:24.075323 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:02:24.075299    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92b375b_7ed0_40e3_8b4e_3024bfe59513.slice/crio-4871828cf614c3dd80ce8aff972cb096dd3ec55f8de6ffbac11a3cef38dbaec7 WatchSource:0}: Error finding container 4871828cf614c3dd80ce8aff972cb096dd3ec55f8de6ffbac11a3cef38dbaec7: Status 404 returned error can't find the container with id 4871828cf614c3dd80ce8aff972cb096dd3ec55f8de6ffbac11a3cef38dbaec7
Sep 16 19:02:24.168138 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:24.168106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerStarted","Data":"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"}
Sep 16 19:02:24.168314 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:24.168196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerStarted","Data":"4871828cf614c3dd80ce8aff972cb096dd3ec55f8de6ffbac11a3cef38dbaec7"}
Sep 16 19:02:28.184587 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:28.184555    2580 generic.go:356] "Generic (PLEG): container finished" podID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerID="8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420" exitCode=0
Sep 16 19:02:28.184958 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:28.184631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerDied","Data":"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"}
Sep 16 19:02:29.190663 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.190634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerStarted","Data":"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"}
Sep 16 19:02:29.191038 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.190693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerStarted","Data":"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"}
Sep 16 19:02:29.191101 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.191039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:29.191185 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.191169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:29.192657 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.192624    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:29.214541 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:29.214493    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podStartSLOduration=2.618028401 podStartE2EDuration="6.214479151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59645075s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:29.213054827 +0000 UTC m=+1079.958550625" watchObservedRunningTime="2026-09-16 19:02:29.214479151 +0000 UTC m=+1079.959974948"
Sep 16 19:02:30.194703 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:30.194659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:31.198957 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:31.198915    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:36.204071 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:36.204042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:02:36.204630 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:36.204605    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:37.245063 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:37.245027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:37.302832 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:37.302804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:46.205536 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:46.205490    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:02:56.205519 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:02:56.205476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:06.204585 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:06.204535    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:16.205293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:16.205239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:26.205523 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:26.205477    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:28.170843 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:28.170811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:33.319836 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.319801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:33.352674 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.352643    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:03:33.356743 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.356724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.359242 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.359222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:03:33.359242 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.359223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:03:33.368711 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.368691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4d8r\" (UniqueName: \"kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.368813 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.368731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.368813 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.368756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.368900 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.368835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.368900 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.368878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.374457 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.374436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:03:33.396100 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.396076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:33.440450 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.440423    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:03:33.444419 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.444404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.446959 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.446941    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:03:33.454107 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.454084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:03:33.469352 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.469434 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4d8r\" (UniqueName: \"kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.469434 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.469434 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.469582 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.469582 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.469582 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.469582 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.469582 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv2v\" (UniqueName: \"kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.469830 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.469830 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:03:33.469645    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:03:33.469830 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:03:33.469778    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls podName:a6b7d34b-cbe6-4284-ba80-29ba5496c3d3 nodeName:}" failed. No retries permitted until 2026-09-16 19:03:33.96975647 +0000 UTC m=+1144.715252252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls") pod "isvc-canary-create-v2-predictor-7855d67596-jrkp9" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:03:33.469830 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.469799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.470221 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.470205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.470260 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.470216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.483255 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.483230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4d8r\" (UniqueName: \"kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.515634 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.515611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:33.570606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.570557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv2v\" (UniqueName: \"kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.570696 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.570617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.570696 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.570649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.570696 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.570685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.570883 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.570733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.571108 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.571091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.571241 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.571224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.571342 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.571321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.573310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.573290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.579247 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.579229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv2v\" (UniqueName: \"kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v\") pod \"isvc-canary-create-predictor-6cc94c6f4f-tffh4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.681168 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.681136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:33.755084 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.755060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:33.973761 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.973725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:33.976442 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:33.976420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-jrkp9\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:34.027710 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:34.027687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:03:34.030029 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:03:34.029999    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d846b0a_0cc7_43c3_a5fd_943b1869b5e4.slice/crio-928c4586c821cab7c4e28cf8a2912574bea7604609165938d777b1d8fa117bb4 WatchSource:0}: Error finding container 928c4586c821cab7c4e28cf8a2912574bea7604609165938d777b1d8fa117bb4: Status 404 returned error can't find the container with id 928c4586c821cab7c4e28cf8a2912574bea7604609165938d777b1d8fa117bb4
Sep 16 19:03:34.269172 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:34.269090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:34.445655 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:34.445621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerStarted","Data":"c69f680a3314318b1cd4dd9fa1c7acc0bc5ba14b1520bdc4c4c34f03b43bd3d0"}
Sep 16 19:03:34.446090 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:34.445678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerStarted","Data":"928c4586c821cab7c4e28cf8a2912574bea7604609165938d777b1d8fa117bb4"}
Sep 16 19:03:34.542775 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:34.542720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:03:34.544891 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:03:34.544866    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b7d34b_cbe6_4284_ba80_29ba5496c3d3.slice/crio-c1fa40d856fa92eb432a6d146d6d98b81d390ae9de66f0f6e4575ef21b5d6ea6 WatchSource:0}: Error finding container c1fa40d856fa92eb432a6d146d6d98b81d390ae9de66f0f6e4575ef21b5d6ea6: Status 404 returned error can't find the container with id c1fa40d856fa92eb432a6d146d6d98b81d390ae9de66f0f6e4575ef21b5d6ea6
Sep 16 19:03:35.451084 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:35.451045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerStarted","Data":"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"}
Sep 16 19:03:35.451522 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:35.451105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerStarted","Data":"c1fa40d856fa92eb432a6d146d6d98b81d390ae9de66f0f6e4575ef21b5d6ea6"}
Sep 16 19:03:36.204686 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:36.204644    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:38.464118 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:38.464089    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerID="c69f680a3314318b1cd4dd9fa1c7acc0bc5ba14b1520bdc4c4c34f03b43bd3d0" exitCode=0
Sep 16 19:03:38.464481 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:38.464161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerDied","Data":"c69f680a3314318b1cd4dd9fa1c7acc0bc5ba14b1520bdc4c4c34f03b43bd3d0"}
Sep 16 19:03:39.470102 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.470068    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerID="bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d" exitCode=0
Sep 16 19:03:39.470522 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.470111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerDied","Data":"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"}
Sep 16 19:03:39.472471 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.472444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerStarted","Data":"a0b9b51f142071ed5b0f72372e17cc25bb04a55517db8a68846b41c14e371e20"}
Sep 16 19:03:39.472570 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.472516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerStarted","Data":"fe8f5c0b1c0f6d513a20d6ff6211f9a57bce0a047a387c6f65b4fe10e65733d8"}
Sep 16 19:03:39.472823 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.472802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:39.472936 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.472876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:39.474318 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.474283    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:39.511329 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:39.511288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podStartSLOduration=2.884770192 podStartE2EDuration="6.51127522s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626505028s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:39.510155415 +0000 UTC m=+1150.255651212" watchObservedRunningTime="2026-09-16 19:03:39.51127522 +0000 UTC m=+1150.256771074"
Sep 16 19:03:40.478365 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.478329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerStarted","Data":"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"}
Sep 16 19:03:40.478822 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.478398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerStarted","Data":"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"}
Sep 16 19:03:40.478822 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.478644    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:40.478822 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.478756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:40.479036 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.478868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:40.480235 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.480213    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:40.503525 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:40.503471    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podStartSLOduration=3.564893698 podStartE2EDuration="7.503457123s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.938563425s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:40.501018317 +0000 UTC m=+1151.246514115" watchObservedRunningTime="2026-09-16 19:03:40.503457123 +0000 UTC m=+1151.248952976"
Sep 16 19:03:41.482836 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:41.482798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:45.483568 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:45.483541    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:03:45.484061 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:45.484035    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:46.206079 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:46.206048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:03:46.487847 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:46.487761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:03:46.488461 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:46.488432    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:53.542315 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:53.542242    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:03:53.542727 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:53.542471    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" containerID="cri-o://079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a" gracePeriod=30
Sep 16 19:03:53.542727 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:53.542527    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kube-rbac-proxy" containerID="cri-o://c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520" gracePeriod=30
Sep 16 19:03:53.572084 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:53.572036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:54.478171 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:54.478141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:54.539670 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:54.539637    2580 generic.go:356] "Generic (PLEG): container finished" podID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerID="c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520" exitCode=2
Sep 16 19:03:54.539828 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:54.539687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerDied","Data":"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"}
Sep 16 19:03:55.484416 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:55.484371    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:03:56.199283 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:56.199239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:03:56.204666 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:56.204629    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 19:03:56.489260 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:56.489168    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:03:58.187346 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.187324    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:03:58.278041 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.277949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrr7m\" (UniqueName: \"kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m\") pod \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") "
Sep 16 19:03:58.278041 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config\") pod \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") "
Sep 16 19:03:58.278041 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls\") pod \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") "
Sep 16 19:03:58.278310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location\") pod \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") "
Sep 16 19:03:58.278310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert\") pod \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\" (UID: \"a92b375b-7ed0-40e3-8b4e-3024bfe59513\") "
Sep 16 19:03:58.278424 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a92b375b-7ed0-40e3-8b4e-3024bfe59513" (UID: "a92b375b-7ed0-40e3-8b4e-3024bfe59513"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:58.278470 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a92b375b-7ed0-40e3-8b4e-3024bfe59513" (UID: "a92b375b-7ed0-40e3-8b4e-3024bfe59513"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:58.278546 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.278525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a92b375b-7ed0-40e3-8b4e-3024bfe59513" (UID: "a92b375b-7ed0-40e3-8b4e-3024bfe59513"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:58.280249 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.280228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m" (OuterVolumeSpecName: "kube-api-access-wrr7m") pod "a92b375b-7ed0-40e3-8b4e-3024bfe59513" (UID: "a92b375b-7ed0-40e3-8b4e-3024bfe59513"). InnerVolumeSpecName "kube-api-access-wrr7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:58.280345 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.280290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a92b375b-7ed0-40e3-8b4e-3024bfe59513" (UID: "a92b375b-7ed0-40e3-8b4e-3024bfe59513"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:58.379083 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.379040    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr7m\" (UniqueName: \"kubernetes.io/projected/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-api-access-wrr7m\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:03:58.379083 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.379078    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:03:58.379083 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.379092    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a92b375b-7ed0-40e3-8b4e-3024bfe59513-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:03:58.379283 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.379105    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a92b375b-7ed0-40e3-8b4e-3024bfe59513-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:03:58.379283 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.379116    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a92b375b-7ed0-40e3-8b4e-3024bfe59513-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:03:58.559191 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.559110    2580 generic.go:356] "Generic (PLEG): container finished" podID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerID="079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a" exitCode=0
Sep 16 19:03:58.559191 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.559168    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"
Sep 16 19:03:58.559436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.559172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerDied","Data":"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"}
Sep 16 19:03:58.559436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.559313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6" event={"ID":"a92b375b-7ed0-40e3-8b4e-3024bfe59513","Type":"ContainerDied","Data":"4871828cf614c3dd80ce8aff972cb096dd3ec55f8de6ffbac11a3cef38dbaec7"}
Sep 16 19:03:58.559436 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.559350    2580 scope.go:117] "RemoveContainer" containerID="c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"
Sep 16 19:03:58.568342 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.568325    2580 scope.go:117] "RemoveContainer" containerID="079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"
Sep 16 19:03:58.575637 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.575620    2580 scope.go:117] "RemoveContainer" containerID="8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"
Sep 16 19:03:58.582180 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.582160    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:03:58.583162 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.583148    2580 scope.go:117] "RemoveContainer" containerID="c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"
Sep 16 19:03:58.583434 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:03:58.583418    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520\": container with ID starting with c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520 not found: ID does not exist" containerID="c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"
Sep 16 19:03:58.583493 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.583441    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520"} err="failed to get container status \"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520\": rpc error: code = NotFound desc = could not find container \"c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520\": container with ID starting with c828971caad285c2dfd9805cd75f2ee449afc9632891fb788d9d0da18d9ed520 not found: ID does not exist"
Sep 16 19:03:58.583493 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.583459    2580 scope.go:117] "RemoveContainer" containerID="079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"
Sep 16 19:03:58.583725 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:03:58.583698    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a\": container with ID starting with 079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a not found: ID does not exist" containerID="079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"
Sep 16 19:03:58.583791 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.583735    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a"} err="failed to get container status \"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a\": rpc error: code = NotFound desc = could not find container \"079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a\": container with ID starting with 079437b882b08888abc04b74c9f9075bec644fc3bb6c1767e1026aac4bcb4b6a not found: ID does not exist"
Sep 16 19:03:58.583791 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.583757    2580 scope.go:117] "RemoveContainer" containerID="8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"
Sep 16 19:03:58.584011 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:03:58.583980    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420\": container with ID starting with 8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420 not found: ID does not exist" containerID="8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"
Sep 16 19:03:58.584049 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.584018    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420"} err="failed to get container status \"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420\": rpc error: code = NotFound desc = could not find container \"8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420\": container with ID starting with 8f330bc16a13192f4f744595f53eb67a538ed329c599e156122e0fc69e186420 not found: ID does not exist"
Sep 16 19:03:58.586590 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:58.586570    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-20b49-predictor-bdb97c8f6-7nll6"]
Sep 16 19:03:59.744440 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:03:59.744401    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" path="/var/lib/kubelet/pods/a92b375b-7ed0-40e3-8b4e-3024bfe59513/volumes"
Sep 16 19:04:05.483995 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:05.483956    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:04:06.488813 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:06.488767    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:04:15.484595 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:15.484555    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:04:16.489095 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:16.489053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:04:25.484461 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:25.484419    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:04:26.489049 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:26.488980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:04:35.484344 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:35.484303    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:04:36.489003 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:36.488958    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:04:45.484821 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:45.484781    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:04:46.489237 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:46.489193    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:04:52.739865 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:52.739834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:04:56.489786 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:04:56.489757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:05:03.467059 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.467027    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:05:03.467514 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.467250    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container" containerID="cri-o://fe8f5c0b1c0f6d513a20d6ff6211f9a57bce0a047a387c6f65b4fe10e65733d8" gracePeriod=30
Sep 16 19:05:03.467514 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.467302    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy" containerID="cri-o://a0b9b51f142071ed5b0f72372e17cc25bb04a55517db8a68846b41c14e371e20" gracePeriod=30
Sep 16 19:05:03.471068 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.471042    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:05:03.471306 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.471269    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" containerID="cri-o://9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8" gracePeriod=30
Sep 16 19:05:03.471432 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.471379    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kube-rbac-proxy" containerID="cri-o://71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5" gracePeriod=30
Sep 16 19:05:03.478504 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.478477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:03.619635 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.619599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:03.818657 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.818579    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerID="a0b9b51f142071ed5b0f72372e17cc25bb04a55517db8a68846b41c14e371e20" exitCode=2
Sep 16 19:05:03.818657 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.818648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerDied","Data":"a0b9b51f142071ed5b0f72372e17cc25bb04a55517db8a68846b41c14e371e20"}
Sep 16 19:05:03.820645 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.820624    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerID="71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5" exitCode=2
Sep 16 19:05:03.820754 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.820669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerDied","Data":"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"}
Sep 16 19:05:03.924889 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:03.924850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:05.479646 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:05.479599    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:05:06.483155 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:06.483114    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 19:05:06.488527 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:06.488498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 19:05:08.515821 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.515794    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:05:08.592396 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.592307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:08.635205 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") pod \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") "
Sep 16 19:05:08.635397 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4d8r\" (UniqueName: \"kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r\") pod \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") "
Sep 16 19:05:08.635397 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config\") pod \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") "
Sep 16 19:05:08.635397 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location\") pod \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") "
Sep 16 19:05:08.635397 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert\") pod \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\" (UID: \"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3\") "
Sep 16 19:05:08.635769 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:08.635858 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:08.635911 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.635889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:08.637573 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.637548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:08.637676 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.637595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r" (OuterVolumeSpecName: "kube-api-access-b4d8r") pod "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" (UID: "a6b7d34b-cbe6-4284-ba80-29ba5496c3d3"). InnerVolumeSpecName "kube-api-access-b4d8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:08.656374 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:05:08.656678 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container"
Sep 16 19:05:08.656724 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656681    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container"
Sep 16 19:05:08.656724 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656724 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656708    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656724 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="storage-initializer"
Sep 16 19:05:08.656724 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656723    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="storage-initializer"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656733    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656743    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="storage-initializer"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656748    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="storage-initializer"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656761    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kserve-container"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656821    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656827    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerName="kube-rbac-proxy"
Sep 16 19:05:08.656881 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.656832    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92b375b-7ed0-40e3-8b4e-3024bfe59513" containerName="kserve-container"
Sep 16 19:05:08.660062 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.660046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.662609 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.662586    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:05:08.662733 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.662659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:05:08.674917 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.674891    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:05:08.682801 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.682773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:08.722124 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.722091    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:05:08.726102 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.726080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.728556 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.728536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:05:08.735282 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.735259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:05:08.736390 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bl6\" (UniqueName: \"kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.736557 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.736651 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.736717 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.736717 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.736828 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736757    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:08.736828 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736774    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:08.736828 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736788    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:08.736969 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736866    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:08.736969 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.736930    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4d8r\" (UniqueName: \"kubernetes.io/projected/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3-kube-api-access-b4d8r\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:08.837946 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.837900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.837946 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.837949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbwfm\" (UniqueName: \"kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.838225 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838225 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838225 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.838348 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838348 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.838348 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bl6\" (UniqueName: \"kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838469 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838469 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.838559 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.838611 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.838567    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:05:08.838668 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.838632    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls podName:8814a47b-9d18-4290-97f3-98d06f629c81 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.338612453 +0000 UTC m=+1240.084108233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls") pod "isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:05:08.838956 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.838885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.839233 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.839214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.846834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.846766    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" containerID="9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8" exitCode=0
Sep 16 19:05:08.846834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.846799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerDied","Data":"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"}
Sep 16 19:05:08.846834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.846830    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"
Sep 16 19:05:08.847087 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.846850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9" event={"ID":"a6b7d34b-cbe6-4284-ba80-29ba5496c3d3","Type":"ContainerDied","Data":"c1fa40d856fa92eb432a6d146d6d98b81d390ae9de66f0f6e4575ef21b5d6ea6"}
Sep 16 19:05:08.847087 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.846876    2580 scope.go:117] "RemoveContainer" containerID="71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"
Sep 16 19:05:08.851861 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.851839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bl6\" (UniqueName: \"kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:08.856509 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.856489    2580 scope.go:117] "RemoveContainer" containerID="9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"
Sep 16 19:05:08.864259 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.864239    2580 scope.go:117] "RemoveContainer" containerID="bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"
Sep 16 19:05:08.870381 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.870356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:05:08.872159 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872127    2580 scope.go:117] "RemoveContainer" containerID="71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"
Sep 16 19:05:08.872452 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.872434    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5\": container with ID starting with 71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5 not found: ID does not exist" containerID="71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"
Sep 16 19:05:08.872534 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872465    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5"} err="failed to get container status \"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5\": rpc error: code = NotFound desc = could not find container \"71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5\": container with ID starting with 71dc3811ebacac715344046b99130e0f84a53cfdeb820debae39dcc1374ad1d5 not found: ID does not exist"
Sep 16 19:05:08.872534 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872488    2580 scope.go:117] "RemoveContainer" containerID="9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"
Sep 16 19:05:08.872534 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872504    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-jrkp9"]
Sep 16 19:05:08.872743 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.872727    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8\": container with ID starting with 9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8 not found: ID does not exist" containerID="9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"
Sep 16 19:05:08.872793 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872749    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8"} err="failed to get container status \"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8\": rpc error: code = NotFound desc = could not find container \"9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8\": container with ID starting with 9ead3039988dc9a2b5ac0e19cb80bc6c4077ff6d237e232c5d8f6d5391e655e8 not found: ID does not exist"
Sep 16 19:05:08.872793 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.872766    2580 scope.go:117] "RemoveContainer" containerID="bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"
Sep 16 19:05:08.873088 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.873066    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d\": container with ID starting with bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d not found: ID does not exist" containerID="bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"
Sep 16 19:05:08.873185 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.873092    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d"} err="failed to get container status \"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d\": rpc error: code = NotFound desc = could not find container \"bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d\": container with ID starting with bee75780a5c289c3c7749d9448410cced94b3aeddbc0c763830bb81e5b9d5c2d not found: ID does not exist"
Sep 16 19:05:08.939926 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.939887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940104 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.939975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940104 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwfm\" (UniqueName: \"kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940104 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940273 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940273 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.940228    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:05:08.940393 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:05:08.940298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls podName:8b80238e-77b4-467f-9073-604b0b9d3720 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:09.440279558 +0000 UTC m=+1240.185775338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls") pod "isvc-canary-promote-predictor-b9b98bcd9-tfzdp" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:05:08.940393 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940615 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.940745 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.940726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.951270 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.951242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwfm\" (UniqueName: \"kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:08.960310 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:08.960283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.222358 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.222328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.343791 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.343756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:09.346292 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.346270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:09.444348 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.444315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:09.446805 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.446781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-tfzdp\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:09.570772 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.570685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:09.637335 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.637309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:09.744576 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.744546    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b7d34b-cbe6-4284-ba80-29ba5496c3d3" path="/var/lib/kubelet/pods/a6b7d34b-cbe6-4284-ba80-29ba5496c3d3/volumes"
Sep 16 19:05:09.848206 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.848179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:05:09.850039 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:05:09.850010    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8814a47b_9d18_4290_97f3_98d06f629c81.slice/crio-55c22516ce3354d2b83dea3e8adf22e4afc3dd08c064dbe3933726e722b11615 WatchSource:0}: Error finding container 55c22516ce3354d2b83dea3e8adf22e4afc3dd08c064dbe3933726e722b11615: Status 404 returned error can't find the container with id 55c22516ce3354d2b83dea3e8adf22e4afc3dd08c064dbe3933726e722b11615
Sep 16 19:05:09.914650 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:09.914462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:05:10.479492 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:10.479444    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:05:10.857629 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:10.857535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerStarted","Data":"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"}
Sep 16 19:05:10.857629 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:10.857599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerStarted","Data":"fe88166e2a2034c1cbe9b8495027a5b7122b9c41034409275a8360ba9cb27bb1"}
Sep 16 19:05:10.859191 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:10.859167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerStarted","Data":"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"}
Sep 16 19:05:10.859297 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:10.859219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerStarted","Data":"55c22516ce3354d2b83dea3e8adf22e4afc3dd08c064dbe3933726e722b11615"}
Sep 16 19:05:11.866308 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:11.866280    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerID="fe8f5c0b1c0f6d513a20d6ff6211f9a57bce0a047a387c6f65b4fe10e65733d8" exitCode=0
Sep 16 19:05:11.866644 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:11.866352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerDied","Data":"fe8f5c0b1c0f6d513a20d6ff6211f9a57bce0a047a387c6f65b4fe10e65733d8"}
Sep 16 19:05:11.910015 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:11.909970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:05:12.064460 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config\") pod \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") "
Sep 16 19:05:12.064460 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktv2v\" (UniqueName: \"kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v\") pod \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") "
Sep 16 19:05:12.064460 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location\") pod \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") "
Sep 16 19:05:12.064460 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls\") pod \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") "
Sep 16 19:05:12.064803 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert\") pod \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\" (UID: \"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4\") "
Sep 16 19:05:12.064852 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" (UID: "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:12.064852 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" (UID: "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:12.064914 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.064855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" (UID: "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:12.066734 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.066714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" (UID: "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:12.066800 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.066770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v" (OuterVolumeSpecName: "kube-api-access-ktv2v") pod "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" (UID: "8d846b0a-0cc7-43c3-a5fd-943b1869b5e4"). InnerVolumeSpecName "kube-api-access-ktv2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:12.165011 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.164959    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.165011 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.165009    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv2v\" (UniqueName: \"kubernetes.io/projected/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kube-api-access-ktv2v\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.165211 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.165024    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.165211 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.165037    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.165211 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.165048    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.872086 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.872052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4" event={"ID":"8d846b0a-0cc7-43c3-a5fd-943b1869b5e4","Type":"ContainerDied","Data":"928c4586c821cab7c4e28cf8a2912574bea7604609165938d777b1d8fa117bb4"}
Sep 16 19:05:12.872086 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.872077    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"
Sep 16 19:05:12.872547 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.872124    2580 scope.go:117] "RemoveContainer" containerID="a0b9b51f142071ed5b0f72372e17cc25bb04a55517db8a68846b41c14e371e20"
Sep 16 19:05:12.881336 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.881320    2580 scope.go:117] "RemoveContainer" containerID="fe8f5c0b1c0f6d513a20d6ff6211f9a57bce0a047a387c6f65b4fe10e65733d8"
Sep 16 19:05:12.888697 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.888683    2580 scope.go:117] "RemoveContainer" containerID="c69f680a3314318b1cd4dd9fa1c7acc0bc5ba14b1520bdc4c4c34f03b43bd3d0"
Sep 16 19:05:12.895031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.895010    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:05:12.897786 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:12.897766    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-tffh4"]
Sep 16 19:05:13.744805 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:13.744779    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" path="/var/lib/kubelet/pods/8d846b0a-0cc7-43c3-a5fd-943b1869b5e4/volumes"
Sep 16 19:05:13.878472 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:13.878443    2580 generic.go:356] "Generic (PLEG): container finished" podID="8814a47b-9d18-4290-97f3-98d06f629c81" containerID="8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f" exitCode=0
Sep 16 19:05:13.878839 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:13.878515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerDied","Data":"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"}
Sep 16 19:05:13.880199 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:13.880179    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b80238e-77b4-467f-9073-604b0b9d3720" containerID="861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1" exitCode=0
Sep 16 19:05:13.880301 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:13.880206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerDied","Data":"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"}
Sep 16 19:05:14.888047 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.888009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerStarted","Data":"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"}
Sep 16 19:05:14.888508 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.888106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerStarted","Data":"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"}
Sep 16 19:05:14.888508 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.888316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:14.890312 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.890288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerStarted","Data":"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"}
Sep 16 19:05:14.890429 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.890333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerStarted","Data":"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"}
Sep 16 19:05:14.890628 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.890610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:14.890673 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.890660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:14.891852 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.891828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:14.911098 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.911050    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podStartSLOduration=3.255998527 podStartE2EDuration="6.911036231s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.655037704s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:14.909843426 +0000 UTC m=+1245.655339223" watchObservedRunningTime="2026-09-16 19:05:14.911036231 +0000 UTC m=+1245.656532088"
Sep 16 19:05:14.933339 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:14.933299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podStartSLOduration=3.111356319 podStartE2EDuration="6.933284834s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821928515s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:14.932748764 +0000 UTC m=+1245.678244562" watchObservedRunningTime="2026-09-16 19:05:14.933284834 +0000 UTC m=+1245.678780665"
Sep 16 19:05:15.895121 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:15.895082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:15.895532 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:15.895125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:15.896276 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:15.896254    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:05:16.900262 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:16.900216    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:05:20.900755 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:20.900675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:05:20.901253 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:20.901223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:21.905469 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:21.905436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:05:21.906434 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:21.906392    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:05:30.902048 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:30.902004    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:31.906838 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:31.906796    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:05:40.901840 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:40.901791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:41.907396 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:41.907352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:05:50.901773 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:50.901734    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:05:51.906749 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:05:51.906706    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:06:00.901872 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:00.901824    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:06:01.907041 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:01.907001    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:06:10.901393 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:10.901338    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:06:11.906679 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:11.906633    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:06:20.901230 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:20.901191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:06:21.906836 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:21.906792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:06:30.902183 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:30.902143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:06:31.907137 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:31.907111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:06:38.846212 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:38.846183    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:06:38.846716 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:38.846411    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" containerID="cri-o://6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f" gracePeriod=30
Sep 16 19:06:38.846716 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:38.846455    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kube-rbac-proxy" containerID="cri-o://8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232" gracePeriod=30
Sep 16 19:06:38.875375 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:38.875341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:39.222463 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:39.222435    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b80238e-77b4-467f-9073-604b0b9d3720" containerID="8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232" exitCode=2
Sep 16 19:06:39.222633 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:39.222505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerDied","Data":"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"}
Sep 16 19:06:41.900893 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:41.900858    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:06:41.907301 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:41.907273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:06:43.090557 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.090535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:06:43.181552 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert\") pod \"8b80238e-77b4-467f-9073-604b0b9d3720\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") "
Sep 16 19:06:43.181552 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location\") pod \"8b80238e-77b4-467f-9073-604b0b9d3720\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") "
Sep 16 19:06:43.181723 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbwfm\" (UniqueName: \"kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm\") pod \"8b80238e-77b4-467f-9073-604b0b9d3720\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") "
Sep 16 19:06:43.181723 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") pod \"8b80238e-77b4-467f-9073-604b0b9d3720\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") "
Sep 16 19:06:43.181723 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config\") pod \"8b80238e-77b4-467f-9073-604b0b9d3720\" (UID: \"8b80238e-77b4-467f-9073-604b0b9d3720\") "
Sep 16 19:06:43.181845 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b80238e-77b4-467f-9073-604b0b9d3720" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:43.181845 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.181828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b80238e-77b4-467f-9073-604b0b9d3720" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:43.182117 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.182100    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.182117 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.182106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b80238e-77b4-467f-9073-604b0b9d3720" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:43.182221 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.182119    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b80238e-77b4-467f-9073-604b0b9d3720-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.183799 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.183780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b80238e-77b4-467f-9073-604b0b9d3720" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:43.183914 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.183898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm" (OuterVolumeSpecName: "kube-api-access-xbwfm") pod "8b80238e-77b4-467f-9073-604b0b9d3720" (UID: "8b80238e-77b4-467f-9073-604b0b9d3720"). InnerVolumeSpecName "kube-api-access-xbwfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:43.242030 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.241980    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b80238e-77b4-467f-9073-604b0b9d3720" containerID="6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f" exitCode=0
Sep 16 19:06:43.242132 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.242059    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"
Sep 16 19:06:43.242132 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.242083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerDied","Data":"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"}
Sep 16 19:06:43.242208 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.242143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp" event={"ID":"8b80238e-77b4-467f-9073-604b0b9d3720","Type":"ContainerDied","Data":"fe88166e2a2034c1cbe9b8495027a5b7122b9c41034409275a8360ba9cb27bb1"}
Sep 16 19:06:43.242208 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.242171    2580 scope.go:117] "RemoveContainer" containerID="8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"
Sep 16 19:06:43.250414 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.250396    2580 scope.go:117] "RemoveContainer" containerID="6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"
Sep 16 19:06:43.257524 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.257509    2580 scope.go:117] "RemoveContainer" containerID="861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"
Sep 16 19:06:43.264441 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.264426    2580 scope.go:117] "RemoveContainer" containerID="8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"
Sep 16 19:06:43.264692 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:06:43.264673    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232\": container with ID starting with 8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232 not found: ID does not exist" containerID="8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"
Sep 16 19:06:43.264763 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.264703    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232"} err="failed to get container status \"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232\": rpc error: code = NotFound desc = could not find container \"8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232\": container with ID starting with 8dbfc32268e13366de47a226393227bee58b8e333a2f25fdfed6cfbb2a255232 not found: ID does not exist"
Sep 16 19:06:43.264763 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.264726    2580 scope.go:117] "RemoveContainer" containerID="6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"
Sep 16 19:06:43.265020 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:06:43.264997    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f\": container with ID starting with 6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f not found: ID does not exist" containerID="6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"
Sep 16 19:06:43.265085 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.265028    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f"} err="failed to get container status \"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f\": rpc error: code = NotFound desc = could not find container \"6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f\": container with ID starting with 6b4d90c825941639d3dd40d1c683dc3041763eb8f2534d1add98b7dad1d2ed4f not found: ID does not exist"
Sep 16 19:06:43.265085 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.265048    2580 scope.go:117] "RemoveContainer" containerID="861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"
Sep 16 19:06:43.265310 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:06:43.265294    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1\": container with ID starting with 861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1 not found: ID does not exist" containerID="861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"
Sep 16 19:06:43.265361 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.265315    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1"} err="failed to get container status \"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1\": rpc error: code = NotFound desc = could not find container \"861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1\": container with ID starting with 861a0a075e5cb2f6d9fe2cff171ab68155f804231e164246a4ea1c7b1bc6fdf1 not found: ID does not exist"
Sep 16 19:06:43.268118 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.268096    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:06:43.270713 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.270695    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tfzdp"]
Sep 16 19:06:43.283433 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.283413    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbwfm\" (UniqueName: \"kubernetes.io/projected/8b80238e-77b4-467f-9073-604b0b9d3720-kube-api-access-xbwfm\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.283433 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.283432    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b80238e-77b4-467f-9073-604b0b9d3720-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.283532 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.283443    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b80238e-77b4-467f-9073-604b0b9d3720-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.744524 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:06:43.744494    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" path="/var/lib/kubelet/pods/8b80238e-77b4-467f-9073-604b0b9d3720/volumes"
Sep 16 19:07:28.409572 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:07:28.409538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:08.979111 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:08.979081    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:08:08.979641 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:08.979313    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" containerID="cri-o://baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99" gracePeriod=30
Sep 16 19:08:08.979641 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:08.979361    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kube-rbac-proxy" containerID="cri-o://4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205" gracePeriod=30
Sep 16 19:08:09.003551 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:09.003521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:09.573865 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:09.573833    2580 generic.go:356] "Generic (PLEG): container finished" podID="8814a47b-9d18-4290-97f3-98d06f629c81" containerID="4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205" exitCode=2
Sep 16 19:08:09.574055 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:09.573884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerDied","Data":"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"}
Sep 16 19:08:10.895726 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:10.895679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:08:10.901543 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:10.901513    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:08:10.902575 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:10.902551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:11.017114 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:11.017079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:13.426054 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.426030    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:08:13.553519 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert\") pod \"8814a47b-9d18-4290-97f3-98d06f629c81\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") "
Sep 16 19:08:13.553519 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location\") pod \"8814a47b-9d18-4290-97f3-98d06f629c81\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") "
Sep 16 19:08:13.553519 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config\") pod \"8814a47b-9d18-4290-97f3-98d06f629c81\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") "
Sep 16 19:08:13.553798 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") pod \"8814a47b-9d18-4290-97f3-98d06f629c81\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") "
Sep 16 19:08:13.553798 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6bl6\" (UniqueName: \"kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6\") pod \"8814a47b-9d18-4290-97f3-98d06f629c81\" (UID: \"8814a47b-9d18-4290-97f3-98d06f629c81\") "
Sep 16 19:08:13.553798 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8814a47b-9d18-4290-97f3-98d06f629c81" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:13.553937 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8814a47b-9d18-4290-97f3-98d06f629c81" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:13.553937 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.553855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8814a47b-9d18-4290-97f3-98d06f629c81" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:13.555817 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.555797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8814a47b-9d18-4290-97f3-98d06f629c81" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:13.555930 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.555911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6" (OuterVolumeSpecName: "kube-api-access-t6bl6") pod "8814a47b-9d18-4290-97f3-98d06f629c81" (UID: "8814a47b-9d18-4290-97f3-98d06f629c81"). InnerVolumeSpecName "kube-api-access-t6bl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:13.590508 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.590480    2580 generic.go:356] "Generic (PLEG): container finished" podID="8814a47b-9d18-4290-97f3-98d06f629c81" containerID="baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99" exitCode=0
Sep 16 19:08:13.590601 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.590517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerDied","Data":"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"}
Sep 16 19:08:13.590601 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.590542    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"
Sep 16 19:08:13.590683 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.590613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp" event={"ID":"8814a47b-9d18-4290-97f3-98d06f629c81","Type":"ContainerDied","Data":"55c22516ce3354d2b83dea3e8adf22e4afc3dd08c064dbe3933726e722b11615"}
Sep 16 19:08:13.590683 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.590654    2580 scope.go:117] "RemoveContainer" containerID="4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"
Sep 16 19:08:13.598814 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.598797    2580 scope.go:117] "RemoveContainer" containerID="baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"
Sep 16 19:08:13.605872 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.605851    2580 scope.go:117] "RemoveContainer" containerID="8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"
Sep 16 19:08:13.612772 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.612757    2580 scope.go:117] "RemoveContainer" containerID="4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"
Sep 16 19:08:13.613036 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:08:13.613020    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205\": container with ID starting with 4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205 not found: ID does not exist" containerID="4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"
Sep 16 19:08:13.613083 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.613043    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205"} err="failed to get container status \"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205\": rpc error: code = NotFound desc = could not find container \"4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205\": container with ID starting with 4a4e73339f12a1c5236c006d1030c2cf615987af47f83f52ae0e1b3a691ea205 not found: ID does not exist"
Sep 16 19:08:13.613083 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.613061    2580 scope.go:117] "RemoveContainer" containerID="baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"
Sep 16 19:08:13.613297 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:08:13.613280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99\": container with ID starting with baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99 not found: ID does not exist" containerID="baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"
Sep 16 19:08:13.613344 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.613303    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99"} err="failed to get container status \"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99\": rpc error: code = NotFound desc = could not find container \"baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99\": container with ID starting with baa35d19a2b0e83d31d4d791d2d7a4b38263b576977e3a968b5ebc7dc1964f99 not found: ID does not exist"
Sep 16 19:08:13.613344 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.613318    2580 scope.go:117] "RemoveContainer" containerID="8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"
Sep 16 19:08:13.613574 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:08:13.613556    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f\": container with ID starting with 8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f not found: ID does not exist" containerID="8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"
Sep 16 19:08:13.613630 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.613582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f"} err="failed to get container status \"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f\": rpc error: code = NotFound desc = could not find container \"8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f\": container with ID starting with 8b4bb68c717369c00b49106726728e2a6c8a96422d5ad3d54eee5a4b5f8fd56f not found: ID does not exist"
Sep 16 19:08:13.622028 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.622009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:08:13.625822 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.625801    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-lq2sp"]
Sep 16 19:08:13.654740 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.654717    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-cabundle-cert\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:08:13.654825 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.654740    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8814a47b-9d18-4290-97f3-98d06f629c81-kserve-provision-location\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:08:13.654825 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.654755    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8814a47b-9d18-4290-97f3-98d06f629c81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:08:13.654825 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.654769    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8814a47b-9d18-4290-97f3-98d06f629c81-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:08:13.654825 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.654782    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bl6\" (UniqueName: \"kubernetes.io/projected/8814a47b-9d18-4290-97f3-98d06f629c81-kube-api-access-t6bl6\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:08:13.744206 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:13.744175    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" path="/var/lib/kubelet/pods/8814a47b-9d18-4290-97f3-98d06f629c81/volumes"
Sep 16 19:08:23.110186 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.110086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:23.151785 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.151745    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:08:23.152140 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152123    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152140 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152155    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152165    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152199    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152217    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152226    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152244    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kube-rbac-proxy"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152264    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152272    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152281    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="storage-initializer"
Sep 16 19:08:23.152293 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152300    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152308    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152406    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kserve-container"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152419    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kube-rbac-proxy"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kube-rbac-proxy"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152436    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b80238e-77b4-467f-9073-604b0b9d3720" containerName="kube-rbac-proxy"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152448    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d846b0a-0cc7-43c3-a5fd-943b1869b5e4" containerName="kserve-container"
Sep 16 19:08:23.153031 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.152457    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8814a47b-9d18-4290-97f3-98d06f629c81" containerName="kserve-container"
Sep 16 19:08:23.157159 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.157138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.162557 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.162533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:08:23.162794 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.162779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:23.163714 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.163692    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:23.163827 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.163697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zmsvt\""
Sep 16 19:08:23.163827 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.163738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 16 19:08:23.168160 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.168138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:08:23.222440 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.222414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.222549 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.222446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.222549 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.222467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rqh\" (UniqueName: \"kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.323597 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.323570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.323716 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.323601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.323716 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.323625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rqh\" (UniqueName: \"kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.323716 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:08:23.323703    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 16 19:08:23.323885 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:08:23.323803    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls podName:09e71baa-8449-421d-b38f-16c2840d280e nodeName:}" failed. No retries permitted until 2026-09-16 19:08:23.823784033 +0000 UTC m=+1434.569279821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-s65nj" (UID: "09e71baa-8449-421d-b38f-16c2840d280e") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 16 19:08:23.324268 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.324246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.335069 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.335039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rqh\" (UniqueName: \"kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.439814 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.439777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:23.827275 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.827190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:23.829722 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:23.829698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-s65nj\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:24.069117 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.069075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:24.338393 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.338371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:08:24.340750 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:08:24.340723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e71baa_8449_421d_b38f_16c2840d280e.slice/crio-cbe081ed4e053880c46fabdd841753270a9b69cdb70eab6f5fd64c55ddfd5e5e WatchSource:0}: Error finding container cbe081ed4e053880c46fabdd841753270a9b69cdb70eab6f5fd64c55ddfd5e5e: Status 404 returned error can't find the container with id cbe081ed4e053880c46fabdd841753270a9b69cdb70eab6f5fd64c55ddfd5e5e
Sep 16 19:08:24.635654 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.635622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerStarted","Data":"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"}
Sep 16 19:08:24.635834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.635680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerStarted","Data":"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"}
Sep 16 19:08:24.635834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.635702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerStarted","Data":"cbe081ed4e053880c46fabdd841753270a9b69cdb70eab6f5fd64c55ddfd5e5e"}
Sep 16 19:08:24.635834 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.635807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:24.635997 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.635874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:24.637393 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.637371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:24.653855 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:24.653809    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" podStartSLOduration=1.653795726 podStartE2EDuration="1.653795726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:24.652178418 +0000 UTC m=+1435.397674214" watchObservedRunningTime="2026-09-16 19:08:24.653795726 +0000 UTC m=+1435.399291523"
Sep 16 19:08:31.648676 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:31.648635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:08:33.127358 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:33.127325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:33.361910 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:08:33.361879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:08.197866 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.197831    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-s65nj_09e71baa-8449-421d-b38f-16c2840d280e/kserve-container/0.log"
Sep 16 19:10:08.439533 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.439499    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:10:08.439764 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.439738    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kserve-container" containerID="cri-o://1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306" gracePeriod=30
Sep 16 19:10:08.439839 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.439775    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kube-rbac-proxy" containerID="cri-o://08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435" gracePeriod=30
Sep 16 19:10:08.691858 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.691835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:10:08.788494 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.788391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config\") pod \"09e71baa-8449-421d-b38f-16c2840d280e\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") "
Sep 16 19:10:08.788494 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.788439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") pod \"09e71baa-8449-421d-b38f-16c2840d280e\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") "
Sep 16 19:10:08.788494 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.788469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46rqh\" (UniqueName: \"kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh\") pod \"09e71baa-8449-421d-b38f-16c2840d280e\" (UID: \"09e71baa-8449-421d-b38f-16c2840d280e\") "
Sep 16 19:10:08.788806 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.788776    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09e71baa-8449-421d-b38f-16c2840d280e" (UID: "09e71baa-8449-421d-b38f-16c2840d280e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:08.790589 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.790566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e71baa-8449-421d-b38f-16c2840d280e" (UID: "09e71baa-8449-421d-b38f-16c2840d280e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:08.790773 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.790754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh" (OuterVolumeSpecName: "kube-api-access-46rqh") pod "09e71baa-8449-421d-b38f-16c2840d280e" (UID: "09e71baa-8449-421d-b38f-16c2840d280e"). InnerVolumeSpecName "kube-api-access-46rqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:08.889746 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.889717    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e71baa-8449-421d-b38f-16c2840d280e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:10:08.889746 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.889741    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e71baa-8449-421d-b38f-16c2840d280e-proxy-tls\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:10:08.889746 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:08.889752    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46rqh\" (UniqueName: \"kubernetes.io/projected/09e71baa-8449-421d-b38f-16c2840d280e-kube-api-access-46rqh\") on node \"ip-10-0-128-253.ec2.internal\" DevicePath \"\""
Sep 16 19:10:09.025734 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025703    2580 generic.go:356] "Generic (PLEG): container finished" podID="09e71baa-8449-421d-b38f-16c2840d280e" containerID="08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435" exitCode=2
Sep 16 19:10:09.025734 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025728    2580 generic.go:356] "Generic (PLEG): container finished" podID="09e71baa-8449-421d-b38f-16c2840d280e" containerID="1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306" exitCode=2
Sep 16 19:10:09.025913 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerDied","Data":"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"}
Sep 16 19:10:09.025913 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025811    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"
Sep 16 19:10:09.025913 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025827    2580 scope.go:117] "RemoveContainer" containerID="08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"
Sep 16 19:10:09.026066 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.025816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerDied","Data":"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"}
Sep 16 19:10:09.026066 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.026008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj" event={"ID":"09e71baa-8449-421d-b38f-16c2840d280e","Type":"ContainerDied","Data":"cbe081ed4e053880c46fabdd841753270a9b69cdb70eab6f5fd64c55ddfd5e5e"}
Sep 16 19:10:09.034616 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.034599    2580 scope.go:117] "RemoveContainer" containerID="1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"
Sep 16 19:10:09.041713 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.041697    2580 scope.go:117] "RemoveContainer" containerID="08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"
Sep 16 19:10:09.041976 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:10:09.041959    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435\": container with ID starting with 08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435 not found: ID does not exist" containerID="08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"
Sep 16 19:10:09.042037 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042006    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"} err="failed to get container status \"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435\": rpc error: code = NotFound desc = could not find container \"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435\": container with ID starting with 08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435 not found: ID does not exist"
Sep 16 19:10:09.042037 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042022    2580 scope.go:117] "RemoveContainer" containerID="1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"
Sep 16 19:10:09.042256 ip-10-0-128-253 kubenswrapper[2580]: E0916 19:10:09.042239    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306\": container with ID starting with 1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306 not found: ID does not exist" containerID="1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"
Sep 16 19:10:09.042302 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042263    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"} err="failed to get container status \"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306\": rpc error: code = NotFound desc = could not find container \"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306\": container with ID starting with 1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306 not found: ID does not exist"
Sep 16 19:10:09.042302 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042276    2580 scope.go:117] "RemoveContainer" containerID="08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"
Sep 16 19:10:09.042507 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042489    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435"} err="failed to get container status \"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435\": rpc error: code = NotFound desc = could not find container \"08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435\": container with ID starting with 08cda3db339786a5ef12d6f797f3061d54a4fbf67fb767b2f0a1a85079af8435 not found: ID does not exist"
Sep 16 19:10:09.042507 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042507    2580 scope.go:117] "RemoveContainer" containerID="1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"
Sep 16 19:10:09.042730 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.042712    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306"} err="failed to get container status \"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306\": rpc error: code = NotFound desc = could not find container \"1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306\": container with ID starting with 1dbc63b0f13e262946cfd2bbc6f85185918b54478935d53b1f145cae8265a306 not found: ID does not exist"
Sep 16 19:10:09.055999 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.055957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:10:09.060118 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.060100    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-s65nj"]
Sep 16 19:10:09.749364 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.749333    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e71baa-8449-421d-b38f-16c2840d280e" path="/var/lib/kubelet/pods/09e71baa-8449-421d-b38f-16c2840d280e/volumes"
Sep 16 19:10:09.766218 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.766194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:09.844221 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:09.844192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.043704 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:40.043662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.232001 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:40.231943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.314518 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:40.314454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:43.076944 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:43.076920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6nq5t_e9f788b9-e314-4ef7-ab91-cf007fc45061/global-pull-secret-syncer/0.log"
Sep 16 19:10:43.169442 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:43.169418    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log"
Sep 16 19:10:43.259030 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:43.259005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-253.ec2.internal_6a2fa8135ce3bbe90bb18a946454dc57/haproxy/0.log"
Sep 16 19:10:46.295476 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:46.295445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:46.424037 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:46.424002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.258792 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:48.258758    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-xkdm4_ab59f219-af81-45d3-a712-d04595c124d7/cluster-monitoring-operator/0.log"
Sep 16 19:10:48.543913 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:48.543825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-pmf4s_9a774927-7dad-4469-81b7-b3495fb9248d/monitoring-plugin/0.log"
Sep 16 19:10:48.593662 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:48.593632    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/node-exporter/0.log"
Sep 16 19:10:48.643106 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:48.643082    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/kube-rbac-proxy/0.log"
Sep 16 19:10:48.686771 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:48.686745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4h7wb_77087871-2210-4cae-b997-e8571af46130/init-textfile/0.log"
Sep 16 19:10:49.408534 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.408508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftd65_ed3d6ac7-7893-4b5d-b0f3-ad05260e4556/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:49.484284 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.484261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ghc6d_409662f8-ba71-4cd2-9433-7529b74575eb/prometheus-operator/0.log"
Sep 16 19:10:49.505755 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.505733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ghc6d_409662f8-ba71-4cd2-9433-7529b74575eb/kube-rbac-proxy/0.log"
Sep 16 19:10:49.670347 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.670284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/thanos-query/0.log"
Sep 16 19:10:49.697658 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.697635    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-web/0.log"
Sep 16 19:10:49.723774 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.723750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy/0.log"
Sep 16 19:10:49.749620 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.749601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/prom-label-proxy/0.log"
Sep 16 19:10:49.778847 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.778832    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-rules/0.log"
Sep 16 19:10:49.806319 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:49.806298    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7668b47f48-7bpg2_d0029a21-2249-44fb-8737-a7b9f5032d6a/kube-rbac-proxy-metrics/0.log"
Sep 16 19:10:50.930214 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"]
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930477    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kserve-container"
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930491    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kserve-container"
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930514    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kube-rbac-proxy"
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930520    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kube-rbac-proxy"
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930574    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kserve-container"
Sep 16 19:10:50.930606 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.930583    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e71baa-8449-421d-b38f-16c2840d280e" containerName="kube-rbac-proxy"
Sep 16 19:10:50.934538 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.934517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:50.937155 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.937127    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txqpj\"/\"kube-root-ca.crt\""
Sep 16 19:10:50.937260 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.937171    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txqpj\"/\"openshift-service-ca.crt\""
Sep 16 19:10:50.938013 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.937998    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-txqpj\"/\"default-dockercfg-hx8r2\""
Sep 16 19:10:50.946090 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:50.946072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"]
Sep 16 19:10:51.096088 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.096054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlr2\" (UniqueName: \"kubernetes.io/projected/42c76949-5d60-4a61-b4fd-f9df08490dbd-kube-api-access-qmlr2\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.096246 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.096114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-lib-modules\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.096246 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.096146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-sys\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.096246 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.096164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-podres\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.096246 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.096195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-proc\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197379 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-lib-modules\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197379 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-sys\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197379 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-podres\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-sys\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-proc\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlr2\" (UniqueName: \"kubernetes.io/projected/42c76949-5d60-4a61-b4fd-f9df08490dbd-kube-api-access-qmlr2\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-lib-modules\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-podres\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.197579 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.197572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42c76949-5d60-4a61-b4fd-f9df08490dbd-proc\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.206345 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.206323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlr2\" (UniqueName: \"kubernetes.io/projected/42c76949-5d60-4a61-b4fd-f9df08490dbd-kube-api-access-qmlr2\") pod \"perf-node-gather-daemonset-98n45\" (UID: \"42c76949-5d60-4a61-b4fd-f9df08490dbd\") " pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.244407 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.244381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:51.514303 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.514219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"]
Sep 16 19:10:51.517163 ip-10-0-128-253 kubenswrapper[2580]: W0916 19:10:51.517136    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42c76949_5d60_4a61_b4fd_f9df08490dbd.slice/crio-71dac1a58391d15a677d69d4b082f33459ee4ed3702b116c918c6ac9094733a9 WatchSource:0}: Error finding container 71dac1a58391d15a677d69d4b082f33459ee4ed3702b116c918c6ac9094733a9: Status 404 returned error can't find the container with id 71dac1a58391d15a677d69d4b082f33459ee4ed3702b116c918c6ac9094733a9
Sep 16 19:10:51.884326 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:51.884302    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64965fdb5c-6c4pd_e15ba382-4195-4a10-9c3d-5462dfd990b7/console/0.log"
Sep 16 19:10:52.182372 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:52.182285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45" event={"ID":"42c76949-5d60-4a61-b4fd-f9df08490dbd","Type":"ContainerStarted","Data":"172dcb3a3991843827c681748a36e85a1396be446ac3a9dbee3e5e9a7b02ae76"}
Sep 16 19:10:52.182372 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:52.182368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45" event={"ID":"42c76949-5d60-4a61-b4fd-f9df08490dbd","Type":"ContainerStarted","Data":"71dac1a58391d15a677d69d4b082f33459ee4ed3702b116c918c6ac9094733a9"}
Sep 16 19:10:52.182745 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:52.182406    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:52.201925 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:52.201877    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45" podStartSLOduration=2.20186184 podStartE2EDuration="2.20186184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:52.201063623 +0000 UTC m=+1582.946559435" watchObservedRunningTime="2026-09-16 19:10:52.20186184 +0000 UTC m=+1582.947357648"
Sep 16 19:10:52.538086 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:52.538017    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-nwjbp_1b10af34-ebe5-4e40-9cbd-e2497f30c59c/volume-data-source-validator/0.log"
Sep 16 19:10:53.622733 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:53.622710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8jt7_475f96b1-e7c0-40b7-8ba0-a49b9c652fe3/dns/0.log"
Sep 16 19:10:53.654207 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:53.654180    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8jt7_475f96b1-e7c0-40b7-8ba0-a49b9c652fe3/kube-rbac-proxy/0.log"
Sep 16 19:10:53.742740 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:53.742713    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bf6tv_317d9c48-0c40-4699-ab85-78fe0f29a92a/dns-node-resolver/0.log"
Sep 16 19:10:54.259539 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:54.259508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59ccf65b59-t67xl_988fe7cd-4b38-44b3-981a-0bfa8dcffd8a/registry/0.log"
Sep 16 19:10:54.345879 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:54.345852    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zfpxr_e1506a9e-c134-411b-8a43-0ff0bd857d2c/node-ca/0.log"
Sep 16 19:10:55.178996 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:55.178968    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6469ffddb8-xx5ck_d23aa9cd-9131-483a-994e-483cbec4153f/router/0.log"
Sep 16 19:10:55.548786 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:55.548718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2dvs4_53030e7e-f330-4608-9405-184801d5a706/serve-healthcheck-canary/0.log"
Sep 16 19:10:56.013570 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:56.013543    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fshll_0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61/kube-rbac-proxy/0.log"
Sep 16 19:10:56.034710 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:56.034692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fshll_0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61/exporter/0.log"
Sep 16 19:10:56.057206 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:56.057184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fshll_0d6eb8bd-e1bd-4c5e-ac82-31d352b8da61/extractor/0.log"
Sep 16 19:10:58.196116 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:58.196089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-txqpj/perf-node-gather-daemonset-98n45"
Sep 16 19:10:58.746082 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:58.746051    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-jvt66_983871e1-653f-4287-a885-a2a9eed1c678/manager/0.log"
Sep 16 19:10:58.971301 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:10:58.971269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vgf7s_1b9e5d8c-14c4-4e9b-b8a1-143500231825/seaweedfs/0.log"
Sep 16 19:11:05.154062 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.154031    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:11:05.176960 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.176931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/egress-router-binary-copy/0.log"
Sep 16 19:11:05.200357 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.200335    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/cni-plugins/0.log"
Sep 16 19:11:05.225668 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.225642    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/bond-cni-plugin/0.log"
Sep 16 19:11:05.251580 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.251564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/routeoverride-cni/0.log"
Sep 16 19:11:05.276870 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.276848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/whereabouts-cni-bincopy/0.log"
Sep 16 19:11:05.300942 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.300918    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjsrm_6b0b751f-bb0a-42fc-927a-754529fd0400/whereabouts-cni/0.log"
Sep 16 19:11:05.823972 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:05.823941    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ghj9s_06ca552f-d354-4b4f-9f3a-a30781b8bd71/kube-multus/0.log"
Sep 16 19:11:06.030844 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:06.030817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svtlg_d79c774c-df1c-4c16-bb55-a4a481ab692b/network-metrics-daemon/0.log"
Sep 16 19:11:06.066354 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:06.066307    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svtlg_d79c774c-df1c-4c16-bb55-a4a481ab692b/kube-rbac-proxy/0.log"
Sep 16 19:11:07.436368 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.436341    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/ovn-controller/0.log"
Sep 16 19:11:07.538742 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.538715    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/ovn-acl-logging/0.log"
Sep 16 19:11:07.565342 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.565318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/kube-rbac-proxy-node/0.log"
Sep 16 19:11:07.597415 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.597395    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:11:07.624978 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.624963    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/northd/0.log"
Sep 16 19:11:07.650390 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.650372    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/nbdb/0.log"
Sep 16 19:11:07.676553 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.676534    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/sbdb/0.log"
Sep 16 19:11:07.715052 ip-10-0-128-253 kubenswrapper[2580]: I0916 19:11:07.714950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqmkz_39723fd9-72ef-45a2-8c31-54d4384ee595/ovnkube-controller/0.log"