Sep 05 02:15:47.388428 ip-10-0-131-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:15:47.388440 ip-10-0-131-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:15:47.388449 ip-10-0-131-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:15:47.388756 ip-10-0-131-142 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:15:58.658872 ip-10-0-131-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:15:58.658887 ip-10-0-131-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9380de20b5e44b9b94a3b75ee5abc66c --
Sep 05 02:17:45.549134 ip-10-0-131-142 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:46.011622 ip-10-0-131-142 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:46.011622 ip-10-0-131-142 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:46.011622 ip-10-0-131-142 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:46.011622 ip-10-0-131-142 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:46.011622 ip-10-0-131-142 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:46.012738 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.012654    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:46.017015 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.016998    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.017015 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017015    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017018    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017023    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017026    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017029    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017032    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017035    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017037    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017040    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017043    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017045    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017048    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017050    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017053    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017055    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017058    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017060    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017062    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017065    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017067    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.017080 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017073    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017075    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017078    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017081    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017084    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017086    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017089    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017092    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017095    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017097    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017100    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017102    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017105    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017107    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017110    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017112    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017114    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017117    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017119    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.017555 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017123    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017127    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017129    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017132    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017135    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017137    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017139    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017142    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017144    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017147    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017149    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017152    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017154    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017156    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017159    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017164    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017167    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017169    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017172    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017175    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.018037 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017177    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017180    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017182    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017186    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017188    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017191    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017195    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017198    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017201    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017204    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017207    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017209    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017212    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017214    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017217    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017219    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017223    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017225    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017227    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.018560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017230    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017232    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017235    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017237    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017240    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017242    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017244    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017247    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017251    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017255    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017258    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017261    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017263    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017266    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017269    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017271    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017274    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017276    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017279    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.017281    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.019023 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018064    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018076    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018080    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018083    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018086    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018089    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018092    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018094    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018097    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018101    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018103    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018106    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018108    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018111    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018114    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018116    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018119    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018122    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018125    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018128    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.019496 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018131    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018136    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018139    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018141    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018144    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018147    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018149    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018152    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018154    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018157    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018159    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018162    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018164    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018167    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018170    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018172    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018175    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018178    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018181    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018183    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.020007 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018186    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018189    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018191    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018194    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018196    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018199    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018201    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018203    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018206    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018208    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018211    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018213    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018217    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018220    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018223    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018226    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018229    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018232    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018236    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.020498 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018239    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018241    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018244    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018247    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018250    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018253    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018257    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018260    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018262    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018265    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018267    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018270    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018273    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018275    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018278    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018280    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018283    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018285    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018288    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018291    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.020983 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018293    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018296    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018298    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018301    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018303    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018306    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018308    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018312    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018314    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018317    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018319    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018323    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018325    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018328    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018331    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018334    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018336    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018339    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018341    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018345    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.021481 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018743    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018748    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018752    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018755    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018773    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018777    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018781    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018785    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018789    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018792    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018794    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018797    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018800    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018803    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018805    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018808    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018810    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018813    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018815    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.021990 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018818    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018820    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018823    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018825    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018829    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018832    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018834    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018838    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018840    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018843    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018845    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018848    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018851    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018854    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018857    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018859    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018862    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018865    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018867    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018870    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.022443 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018873    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018877    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018880    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018883    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018887    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018891    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018894    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018896    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018899    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018901    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018904    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018906    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018910    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018912    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018915    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018917    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018920    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018923    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018926    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018929    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.023273 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018932    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018934    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018936    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018939    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018941    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018944    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018946    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018948    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018951    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018953    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018956    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018958    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018960    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018963    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018965    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018967    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018970    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018972    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018975    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018977    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.024014 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018980    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018982    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018984    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018987    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018989    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018992    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018994    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018996    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.018999    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019002    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019005    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019008    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019010    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019013    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019015    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019018    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019020    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019022    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019025    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019027    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.024511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019098    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019136    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019142    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019147    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019151    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019154    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019158    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019162    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019165    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019168    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019172    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019175    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019178    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019181    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019184    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019187    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019190    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019192    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019196    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019199    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019202    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019205    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019208    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019214    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:46.025014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019217    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019220    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019223    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019227    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019229    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019232    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019235    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019238    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019242    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019245    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019248    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019250    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019254    2582 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019257    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019261    2582 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019264    2582 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019267    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019270    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019273    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019276    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019279    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019282    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019285    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019288    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019291    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:46.025591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019293    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019296    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019299    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019302    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019305    2582 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019309    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019315    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019319    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019322    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019325    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019328    2582 flags.go:64] FLAG: --help="false"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019331    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019334    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019336    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019340    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019343    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019346    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019349    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019352    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019355    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019358    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019360    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019363    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019366    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:46.026219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019369    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019371    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019375    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019378    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019381    2582 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019383    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019386    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019389    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019394    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019398    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019400    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019403    2582 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019406    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019409    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019413    2582 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019416    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019421    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019424    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019428    2582 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019431    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019434    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019437    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019440    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019443    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019446    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:46.026822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019448    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019456    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019458    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019462    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019464    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019468    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019473    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019475    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019478    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019483    2582 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019486    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019489    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b6c908cd464ae3a2"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019492    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019495    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019498    2582 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019501    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019504    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019508    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019510    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019513    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019516    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019519    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019523    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019526    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019529    2582 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:46.027564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019532    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019535    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019538    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019541    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019543    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019546    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019549    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019552    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019554    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019557    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019560    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019563    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019566    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019569    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019571    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019577    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019581    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019584    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019587    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019590    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019593    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019595    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019598    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019601    2582 flags.go:64] FLAG: --v="2"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019605    2582 flags.go:64] FLAG: --version="false"
Sep 05 02:17:46.028188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019608    2582 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019612    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.019615    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019751    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019757    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019777    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019782    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019787    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019790    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019793    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019796    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019799    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019801    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019804    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019806    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019809    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019812    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019815    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019817    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019819    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.028799 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019822    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019824    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019827    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019831    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019834    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019836    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019839    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019841    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019844    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019846    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019849    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019851    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019853    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019856    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019859    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019861    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019865    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019867    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019870    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.029296 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019872    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019876    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019878    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019881    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019883    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019886    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019888    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019890    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019893    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019895    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019898    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019900    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019903    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019905    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019908    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019910    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019914    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019917    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019919    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019922    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.029755 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019924    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019927    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019929    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019932    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019934    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019937    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019939    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019941    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019944    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019947    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019950    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019952    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019955    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019957    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019960    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019962    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019965    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019967    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019970    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019973    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.030381 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019977    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019980    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019982    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019986    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019990    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019994    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019996    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.019999    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020004    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020006    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020009    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020012    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020014    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020017    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020020    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020022    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020025    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020027    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020029    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.030926 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020032    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.031400 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020034    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.031400 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020038    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.031400 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.020040    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.031400 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.020779    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:46.031687 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.031661    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:46.031720 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.031688    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:46.032593 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.032582    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:46.032626 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.032594    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:46.032675 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032666    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.032675 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032674    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032680    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032685    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032688    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032691    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032695    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032698    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032701    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032704    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032707    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032710    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032712    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032715    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032718    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032727    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032730    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032733    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032735    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032738    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.032731 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032741    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032744    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032747    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032749    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032752    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032754    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032756    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032773    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032795    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032798    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032803    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032806    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032809    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032812    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032815    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032818    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032821    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032823    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032826    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.033208 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032828    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032831    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032833    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032836    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032838    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032841    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032843    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032847    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032850    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032852    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032855    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032857    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032860    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032862    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032864    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032868    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032871    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032873    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032876    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032879    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.033669 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032881    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032884    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032886    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032888    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032891    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032894    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032896    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032898    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032901    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032903    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032906    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032908    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032911    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032915    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032919    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032921    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032924    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032926    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032929    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032931    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.034252 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032935    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032938    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032941    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032944    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032946    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032949    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032959    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032963    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032967    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032969    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032972    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032974    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032977    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032980    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032982    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032985    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032987    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032990    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032993    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.034739 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.032996    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.033001    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033122    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033127    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033130    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033133    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033137    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033140    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033143    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033146    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033148    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033151    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033153    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033156    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:46.035216 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033159    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033162    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033164    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033167    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033170    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033172    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033175    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033177    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033180    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033183    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033185    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033188    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033190    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033193    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033195    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033198    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033200    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033203    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033206    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:46.035560 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033209    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033211    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033214    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033217    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033219    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033222    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033225    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033227    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033229    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033232    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033235    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033237    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033240    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033243    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033248    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033250    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033253    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033256    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033259    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:46.036032 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033261    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033264    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033268    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033270    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033273    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033276    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033278    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033280    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033283    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033285    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033288    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033291    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033293    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033296    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033299    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033301    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033303    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033306    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033308    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033311    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:46.036493 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033314    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033316    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033318    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033321    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033323    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033326    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033328    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033331    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033334    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033336    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033339    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033341    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033343    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033347    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033349    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033352    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033354    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033357    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033359    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033361    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:46.037000 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033364    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033366    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033369    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033372    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033376    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033379    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033381    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033384    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:46.033386    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.033391    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.033526    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:46.037522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.037062    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:46.038216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.038203    2582 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:46.038327 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.038311    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:46.038356 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.038344    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:46.041277 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.041262    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:46.044427 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.044408    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:46.055461 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.055437    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:46.060096 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.060075    2582 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:46.061974 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.061953    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:46.064651 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.064632    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dca0f639-aed0-466d-a3f1-e3df4538413b:/dev/nvme0n1p4 f05d465f-5801-4849-9859-1e46d541e92a:/dev/nvme0n1p3]
Sep 05 02:17:46.064703 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.064651    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:46.070306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.070154    2582 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:46.068678207 +0000 UTC m=+0.390450732 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096457 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234728c39a26d34dc776bef47375e6 SystemUUID:ec234728-c39a-26d3-4dc7-76bef47375e6 BootID:9380de20-b5e4-4b9b-94a3-b75ee5abc66c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:02:87:ce:95:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:02:87:ce:95:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:f2:69:b7:66:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:46.070306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.070275    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:46.070306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.070287    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:46.070544 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.070391    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:46.072391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.072362    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:46.072563 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.072398    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:46.073309 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.073295    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:46.073362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.073314    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:46.073362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.073325    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:46.074179 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.074167    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:46.076117 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.076105    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:46.080829 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.080811    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:46.081004 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.080991    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:46.081079 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081069    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:46.081131 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081086    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:46.081131 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081108    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:46.081131 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081125    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:46.081241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081137    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:46.081541 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081528    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:46.081613 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.081554    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:46.083985 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.083967    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:46.087211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087196    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:46.087264 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087227    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:46.087391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087381    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:46.087421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087395    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:46.087421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087401    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:46.087421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087407    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:46.087421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087414    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:46.087421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.087420    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:46.088085 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088074    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:46.088126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088086    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:46.088126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088094    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:46.088126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088100    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:46.088126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088114    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:46.088126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.088121    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:46.089219 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.089201    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nm7k8"
Sep 05 02:17:46.090018 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.090005    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:46.090064 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.090020    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:46.092002 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.091980    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:46.092002 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.091988    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:46.094221 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.094209    2582 server.go:1268] "Started kubelet"
Sep 05 02:17:46.094338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.094293    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:46.094400 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.094368    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:46.094446 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.094298    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:46.095102 ip-10-0-131-142 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:46.095786 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.095754    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:46.095928 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.095915    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:46.100550 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.100430    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nm7k8"
Sep 05 02:17:46.101948 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.101912    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:46.102531 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.102516    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:46.103728 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.102561    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-142.ec2.internal.18d24b2ebacbc405  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-142.ec2.internal,UID:ip-10-0-131-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-142.ec2.internal,},FirstTimestamp:2026-09-05 02:17:46.094003205 +0000 UTC m=+0.415775716,LastTimestamp:2026-09-05 02:17:46.094003205 +0000 UTC m=+0.415775716,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-142.ec2.internal,}"
Sep 05 02:17:46.105948 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.105930    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:46.106627 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.106612    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:46.106824 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.106803    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.107414 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.107391    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:46.107509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.107432    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:46.107509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.107447    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:46.107614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.107544    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:46.107614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.107552    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:46.109983 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.109964    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:46.109983 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.109983    2582 factory.go:55] Registering systemd factory
Sep 05 02:17:46.110123 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.109992    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:46.110257 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110241    2582 factory.go:153] Registering CRI-O factory
Sep 05 02:17:46.110304 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110259    2582 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:46.110304 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110272    2582 factory.go:103] Registering Raw factory
Sep 05 02:17:46.110304 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110281    2582 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:46.110434 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110279    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:46.110631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.110617    2582 manager.go:349] Starting recovery of all containers
Sep 05 02:17:46.113781 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.113746    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.119669 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.119654    2582 manager.go:354] Recovery completed
Sep 05 02:17:46.123640 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.123627    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.125440 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.125423    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.125494 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.125460    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.125494 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.125472    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.126094 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126081    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:46.126161 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126095    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:46.126161 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126113    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:46.126243 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126086    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.126952 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126935    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.127033 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126970    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.127033 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.126988    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.128202 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.128186    2582 policy_none.go:49] "None policy: Start"
Sep 05 02:17:46.128268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.128205    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:46.128268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.128218    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:46.130222 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.130209    2582 policy_none.go:47] "Start"
Sep 05 02:17:46.172444 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.172428    2582 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.172561    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.172580    2582 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.172900    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.172914    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.173014    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.173135    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.173144    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.173829    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.173877    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.176211    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.177554    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.177606    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.177631    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.177669    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:46.186859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.179926    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:46.274101 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.273990    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.276417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.276400    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.276482 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.276433    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.276482 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.276443    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.276482 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.276466    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.279249 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.278469    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"]
Sep 05 02:17:46.279580 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.279566    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.281682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.281662    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.281808 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.281697    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.281808 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.281711    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.282949 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.282905    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.282949 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.282926    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-142.ec2.internal\": node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.284106 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284091    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.284216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.284263 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284230    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.284906 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284882    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.284999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284911    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.284999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284925    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.284999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284939    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.284999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284966    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.284999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.284976    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.287208 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.287192    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.287287 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.287216    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:46.288857 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.288832    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:46.288857 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.288855    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:46.288999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.288864    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:46.300814 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.300738    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.307952 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.307932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.308043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.307963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.308043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.307988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.315588 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.315572    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.318974 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.318958    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-142.ec2.internal\" not found" node="ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.401485 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.401449    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.408737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.408854 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.408854 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.408854 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.408982 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58007dd99525496ea22ff71e4a80cd63-config\") pod \"kube-apiserver-proxy-ip-10-0-131-142.ec2.internal\" (UID: \"58007dd99525496ea22ff71e4a80cd63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.408982 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.408888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ee296aad7c4e56d27fc66bd1b97d18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal\" (UID: \"59ee296aad7c4e56d27fc66bd1b97d18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.502475 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.502444    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.603303 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.603221    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.619728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.619706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.622912 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:46.622890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:46.704490 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.704314    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.804824 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.804796    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:46.905357 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:46.905289    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.005867 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.005823    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.038309 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.038282    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:17:47.038698 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.038431    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:47.038698 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.038439    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:47.106451 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.106424    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.107577 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.107548    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:46 +0000 UTC" deadline="2028-05-12 10:54:25.839980362 +0000 UTC"
Sep 05 02:17:47.107639 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.107580    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14768h36m38.732404972s"
Sep 05 02:17:47.163547 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.163479    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:47.207245 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.207216    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.247743 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:47.247696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58007dd99525496ea22ff71e4a80cd63.slice/crio-d32ee3834fb8c0bc74af2995039d940752ee23a5f4ff6d899a0d31be863d19d0 WatchSource:0}: Error finding container d32ee3834fb8c0bc74af2995039d940752ee23a5f4ff6d899a0d31be863d19d0: Status 404 returned error can't find the container with id d32ee3834fb8c0bc74af2995039d940752ee23a5f4ff6d899a0d31be863d19d0
Sep 05 02:17:47.248253 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:47.248229    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee296aad7c4e56d27fc66bd1b97d18.slice/crio-d47752c020cfba2b116cf3084e42f036f62b9e68121aa16b868f7db3401cbf0f WatchSource:0}: Error finding container d47752c020cfba2b116cf3084e42f036f62b9e68121aa16b868f7db3401cbf0f: Status 404 returned error can't find the container with id d47752c020cfba2b116cf3084e42f036f62b9e68121aa16b868f7db3401cbf0f
Sep 05 02:17:47.254204 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.254188    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:17:47.308237 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.308201    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.408658 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.408630    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.509169 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:47.509087    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-142.ec2.internal\" not found"
Sep 05 02:17:47.594266 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.594234    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:47.607535 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.607503    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:47.619119 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.619090    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:47.621509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.621474    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal"
Sep 05 02:17:47.631012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:47.630987    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:48.049620 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.049590    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:48.082615 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.082582    2582 apiserver.go:52] "Watching apiserver"
Sep 05 02:17:48.092477 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.092451    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:17:48.093031 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.092994    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gjfmw","openshift-ovn-kubernetes/ovnkube-node-b72d9","kube-system/global-pull-secret-syncer-w8dhc","kube-system/konnectivity-agent-cplkx","kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kfh4z","openshift-image-registry/node-ca-bh2bf","openshift-multus/multus-bw72b","openshift-multus/network-metrics-daemon-2nhpg","openshift-network-diagnostics/network-check-target-xxl5g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm","openshift-dns/node-resolver-lclfz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-bpws9"]
Sep 05 02:17:48.095752 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.095729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.099045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.098111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.099045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.098338    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:17:48.099045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.098625    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.099045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.098723    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.099045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.098890    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hppp9\""
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.100407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.100517    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.100873    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.100883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.100911    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.101157    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.101592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.101237    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dwnwz\""
Sep 05 02:17:48.102940 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.102916    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.109122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.106295    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jqjw2\""
Sep 05 02:17:48.109122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.106545    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:17:48.109122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.106797    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:48.109122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.107397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.109122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.109114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.111012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.110400    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vrp77\""
Sep 05 02:17:48.111012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.110407    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.111012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.110612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.111012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.110701    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:17:48.111375 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.111314    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:17:48.111864 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.111842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.111958 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.111914    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:48.112342 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.112321    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.113086 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.112869    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.113338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.113316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2lmgp\""
Sep 05 02:17:48.114661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.114337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.117338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.117318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5gjhl\""
Sep 05 02:17:48.117438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.117356    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:17:48.117438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.117390    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:17:48.117693 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.117676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.118322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.118322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd978685-2990-4515-8493-ae261615757d-host-slash\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.118322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-modprobe-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-system-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-socket-dir-parent\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-hostroot\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-multus-certs\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-etc-kubernetes\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwc5k\" (UniqueName: \"kubernetes.io/projected/629c6726-442d-4ee3-99a5-323e477b1e53-kube-api-access-gwc5k\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-k8s-cni-cncf-io\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-multus\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84933ea4-b102-42f9-b2e9-5f47c655130c-konnectivity-ca\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/629c6726-442d-4ee3-99a5-323e477b1e53-serviceca\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-dbus\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-host\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.118764 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-tmp\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-daemon-config\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84933ea4-b102-42f9-b2e9-5f47c655130c-agent-certs\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysconfig\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-conf\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-sys\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.118943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cnibin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-bin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-conf-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629c6726-442d-4ee3-99a5-323e477b1e53-host\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.119157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpdl\" (UniqueName: \"kubernetes.io/projected/bd978685-2990-4515-8493-ae261615757d-kube-api-access-mwpdl\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cni-binary-copy\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-kubernetes\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-systemd\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-lib-modules\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-tuned\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz6h\" (UniqueName: \"kubernetes.io/projected/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-kube-api-access-zvz6h\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnb5\" (UniqueName: \"kubernetes.io/projected/d06cc8bd-5304-4d9b-8737-48a1f5370e22-kube-api-access-fnnb5\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd978685-2990-4515-8493-ae261615757d-iptables-alerter-script\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-run\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-netns\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-kubelet-config\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-var-lib-kubelet\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvc5\" (UniqueName: \"kubernetes.io/projected/37c79c34-5580-45bb-85b2-b82afd3dd6e2-kube-api-access-5cvc5\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-kubelet\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.119682    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:48.120097 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.119747    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:17:48.121498 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.121205    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4j6n\""
Sep 05 02:17:48.121498 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.121244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:17:48.121657 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.121625    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:17:48.122631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.122037    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.122631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.122055    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:17:48.122631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.122273    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.122631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.122320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:17:48.122631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.122321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.124078 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.123974    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:48.124459 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.124425    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:17:48.124592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.124531    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.124652 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.124609    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.124877 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.124860    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xs6k7\""
Sep 05 02:17:48.124934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.124885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.127485 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.127321    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:17:48.127485 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.127342    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:17:48.127485 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.127370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7z7q2\""
Sep 05 02:17:48.143641 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.143614    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ld7rg"
Sep 05 02:17:48.152724 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.152702    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ld7rg"
Sep 05 02:17:48.182647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.182578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" event={"ID":"58007dd99525496ea22ff71e4a80cd63","Type":"ContainerStarted","Data":"d32ee3834fb8c0bc74af2995039d940752ee23a5f4ff6d899a0d31be863d19d0"}
Sep 05 02:17:48.183934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.183903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerStarted","Data":"d47752c020cfba2b116cf3084e42f036f62b9e68121aa16b868f7db3401cbf0f"}
Sep 05 02:17:48.203692 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.203667    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:48.208847 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.208826    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:17:48.220623 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-hostroot\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-multus-certs\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-etc-kubernetes\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-etc-kubernetes\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-hostroot\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.220797 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-multus-certs\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jn4\" (UniqueName: \"kubernetes.io/projected/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kube-api-access-56jn4\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-multus\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cnibin\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-env-overrides\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/629c6726-442d-4ee3-99a5-323e477b1e53-serviceca\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220934    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-multus\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.220953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-dbus\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-tmp\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-log-socket\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysconfig\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-conf\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.221181 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-bin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-dbus\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-conf-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-node-log\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysconfig\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.221316    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.221386    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:48.721361457 +0000 UTC m=+3.043133958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-sysctl-conf\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-cni-bin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/629c6726-442d-4ee3-99a5-323e477b1e53-serviceca\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221475    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-conf-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629c6726-442d-4ee3-99a5-323e477b1e53-host\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629c6726-442d-4ee3-99a5-323e477b1e53-host\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-ovn\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-kubernetes\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222048 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-systemd\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-tuned\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-kubernetes\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-systemd\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-netd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd978685-2990-4515-8493-ae261615757d-iptables-alerter-script\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-run\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-systemd-units\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.221995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-netns\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-run\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-kubelet\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-var-lib-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-modprobe-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-var-lib-kubelet\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-system-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.222796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-modprobe-d\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-system-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-socket-dir-parent\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-config\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwc5k\" (UniqueName: \"kubernetes.io/projected/629c6726-442d-4ee3-99a5-323e477b1e53-kube-api-access-gwc5k\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-socket-dir-parent\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-k8s-cni-cncf-io\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-k8s-cni-cncf-io\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd978685-2990-4515-8493-ae261615757d-iptables-alerter-script\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84933ea4-b102-42f9-b2e9-5f47c655130c-konnectivity-ca\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-host\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-host\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-daemon-config\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84933ea4-b102-42f9-b2e9-5f47c655130c-agent-certs\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-bin\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.223930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58jx\" (UniqueName: \"kubernetes.io/projected/0b1f3509-edfb-416a-b956-b469a9841b91-kube-api-access-x58jx\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-device-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-sys\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cnibin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-system-cni-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-kubelet\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.222982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovn-node-metrics-cert\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpdl\" (UniqueName: \"kubernetes.io/projected/bd978685-2990-4515-8493-ae261615757d-kube-api-access-mwpdl\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cni-binary-copy\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/84933ea4-b102-42f9-b2e9-5f47c655130c-konnectivity-ca\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-sys\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-lib-modules\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz6h\" (UniqueName: \"kubernetes.io/projected/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-kube-api-access-zvz6h\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-daemon-config\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnb5\" (UniqueName: \"kubernetes.io/projected/d06cc8bd-5304-4d9b-8737-48a1f5370e22-kube-api-access-fnnb5\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-etc-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.224806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-lib-modules\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-netns\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2whh\" (UniqueName: \"kubernetes.io/projected/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-kube-api-access-v2whh\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-systemd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-host-run-netns\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-script-lib\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cnibin\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b1f3509-edfb-416a-b956-b469a9841b91-hosts-file\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1f3509-edfb-416a-b956-b469a9841b91-tmp-dir\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d06cc8bd-5304-4d9b-8737-48a1f5370e22-cni-binary-copy\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-kubelet-config\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-var-lib-kubelet\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvc5\" (UniqueName: \"kubernetes.io/projected/37c79c34-5580-45bb-85b2-b82afd3dd6e2-kube-api-access-5cvc5\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.223989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79bd5ce1-cb17-4035-984a-dfc16c53b637-kubelet-config\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-var-lib-kubelet\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.225654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d06cc8bd-5304-4d9b-8737-48a1f5370e22-multus-cni-dir\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-slash\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kslt\" (UniqueName: \"kubernetes.io/projected/552d8eb0-1a34-4f95-af5f-3c863be9df7f-kube-api-access-9kslt\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd978685-2990-4515-8493-ae261615757d-host-slash\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.224406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd978685-2990-4515-8493-ae261615757d-host-slash\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.224421    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.224683    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:48.724580561 +0000 UTC m=+3.046353085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.225838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-etc-tuned\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.226103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/84933ea4-b102-42f9-b2e9-5f47c655130c-agent-certs\") pod \"konnectivity-agent-cplkx\" (UID: \"84933ea4-b102-42f9-b2e9-5f47c655130c\") " pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.226437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.226395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-tmp\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.232147 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.232125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpdl\" (UniqueName: \"kubernetes.io/projected/bd978685-2990-4515-8493-ae261615757d-kube-api-access-mwpdl\") pod \"iptables-alerter-gjfmw\" (UID: \"bd978685-2990-4515-8493-ae261615757d\") " pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.232507 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.232458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwc5k\" (UniqueName: \"kubernetes.io/projected/629c6726-442d-4ee3-99a5-323e477b1e53-kube-api-access-gwc5k\") pod \"node-ca-bh2bf\" (UID: \"629c6726-442d-4ee3-99a5-323e477b1e53\") " pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.233306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.233260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnb5\" (UniqueName: \"kubernetes.io/projected/d06cc8bd-5304-4d9b-8737-48a1f5370e22-kube-api-access-fnnb5\") pod \"multus-bw72b\" (UID: \"d06cc8bd-5304-4d9b-8737-48a1f5370e22\") " pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.233622 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.233588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz6h\" (UniqueName: \"kubernetes.io/projected/18ca5a4f-3782-4a35-9e89-2c34cc0f96b7-kube-api-access-zvz6h\") pod \"tuned-kfh4z\" (UID: \"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7\") " pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.233796 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.233770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvc5\" (UniqueName: \"kubernetes.io/projected/37c79c34-5580-45bb-85b2-b82afd3dd6e2-kube-api-access-5cvc5\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.325041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.324950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-systemd-units\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.324999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-netns\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-netns\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-var-lib-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-systemd-units\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-config\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-var-lib-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.325329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-bin\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58jx\" (UniqueName: \"kubernetes.io/projected/0b1f3509-edfb-416a-b956-b469a9841b91-kube-api-access-x58jx\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-device-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-bin\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-system-cni-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-kubelet\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovn-node-metrics-cert\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-etc-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2whh\" (UniqueName: \"kubernetes.io/projected/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-kube-api-access-v2whh\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-systemd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-script-lib\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b1f3509-edfb-416a-b956-b469a9841b91-hosts-file\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1f3509-edfb-416a-b956-b469a9841b91-tmp-dir\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.325816 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-slash\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kslt\" (UniqueName: \"kubernetes.io/projected/552d8eb0-1a34-4f95-af5f-3c863be9df7f-kube-api-access-9kslt\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-config\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-device-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-system-cni-dir\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-kubelet\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jn4\" (UniqueName: \"kubernetes.io/projected/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kube-api-access-56jn4\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.325994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cnibin\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-env-overrides\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-log-socket\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-node-log\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.326682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-systemd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-ovn\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-netd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-cni-netd\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1f3509-edfb-416a-b956-b469a9841b91-tmp-dir\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b1f3509-edfb-416a-b956-b469a9841b91-hosts-file\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-slash\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-etc-openvswitch\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.327511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-cnibin\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-node-log\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovnkube-script-lib\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-run-ovn\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.326997    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/552d8eb0-1a34-4f95-af5f-3c863be9df7f-log-socket\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.327133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/552d8eb0-1a34-4f95-af5f-3c863be9df7f-env-overrides\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.328153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.327333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.328780 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.328739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/552d8eb0-1a34-4f95-af5f-3c863be9df7f-ovn-node-metrics-cert\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.337640 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.337609    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:48.337640 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.337640    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:48.337847 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.337655    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:48.337847 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.337740    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:48.837701916 +0000 UTC m=+3.159474443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:48.340520 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.340499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58jx\" (UniqueName: \"kubernetes.io/projected/0b1f3509-edfb-416a-b956-b469a9841b91-kube-api-access-x58jx\") pod \"node-resolver-lclfz\" (UID: \"0b1f3509-edfb-416a-b956-b469a9841b91\") " pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.340626 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.340526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jn4\" (UniqueName: \"kubernetes.io/projected/a657a2f5-bf54-4195-a98b-dd5e594ad5e7-kube-api-access-56jn4\") pod \"aws-ebs-csi-driver-node-9cxqm\" (UID: \"a657a2f5-bf54-4195-a98b-dd5e594ad5e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.340679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.340628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kslt\" (UniqueName: \"kubernetes.io/projected/552d8eb0-1a34-4f95-af5f-3c863be9df7f-kube-api-access-9kslt\") pod \"ovnkube-node-b72d9\" (UID: \"552d8eb0-1a34-4f95-af5f-3c863be9df7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.341199 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.341154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2whh\" (UniqueName: \"kubernetes.io/projected/bf8efa68-e2e0-478d-8244-a1433ec4c1a2-kube-api-access-v2whh\") pod \"multus-additional-cni-plugins-bpws9\" (UID: \"bf8efa68-e2e0-478d-8244-a1433ec4c1a2\") " pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.410540 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.410499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bh2bf"
Sep 05 02:17:48.421679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.421649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw72b"
Sep 05 02:17:48.432187 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.432163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:17:48.440141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.440121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gjfmw"
Sep 05 02:17:48.446251 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.446233    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kfh4z"
Sep 05 02:17:48.453144 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.453114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpws9"
Sep 05 02:17:48.462561 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.462537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:17:48.467420 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.467396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm"
Sep 05 02:17:48.473051 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.473025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lclfz"
Sep 05 02:17:48.729715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.729614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:48.729715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.729673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:48.729962 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.729791    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:48.729962 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.729789    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:48.729962 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.729868    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.72984911 +0000 UTC m=+4.051621613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:48.729962 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.729888    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.729877366 +0000 UTC m=+4.051649864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:48.931110 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:48.931062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:48.931309 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.931218    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:48.931309 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.931253    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:48.931309 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.931268    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:48.931427 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:48.931338    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.931318361 +0000 UTC m=+4.253090883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:49.156084 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:49.155984    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:48 +0000 UTC" deadline="2028-06-20 08:28:11.276689702 +0000 UTC"
Sep 05 02:17:49.156084 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:49.156034    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15702h10m22.120658166s"
Sep 05 02:17:49.211261 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.211225    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06cc8bd_5304_4d9b_8737_48a1f5370e22.slice/crio-d3690c824b9ca0bfc3a1f5ad1b8cd8e4dc7272851708eb0df1cf368c167fa70c WatchSource:0}: Error finding container d3690c824b9ca0bfc3a1f5ad1b8cd8e4dc7272851708eb0df1cf368c167fa70c: Status 404 returned error can't find the container with id d3690c824b9ca0bfc3a1f5ad1b8cd8e4dc7272851708eb0df1cf368c167fa70c
Sep 05 02:17:49.213454 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.213413    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84933ea4_b102_42f9_b2e9_5f47c655130c.slice/crio-a77a0ea6a805f33ab553bccaea72297749fd7e94e1079ef0bb89295328b32910 WatchSource:0}: Error finding container a77a0ea6a805f33ab553bccaea72297749fd7e94e1079ef0bb89295328b32910: Status 404 returned error can't find the container with id a77a0ea6a805f33ab553bccaea72297749fd7e94e1079ef0bb89295328b32910
Sep 05 02:17:49.216146 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.216070    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ca5a4f_3782_4a35_9e89_2c34cc0f96b7.slice/crio-febe5b190446c8d5fa19f18b39ebb985b42f852634a46116601f4dcae7133450 WatchSource:0}: Error finding container febe5b190446c8d5fa19f18b39ebb985b42f852634a46116601f4dcae7133450: Status 404 returned error can't find the container with id febe5b190446c8d5fa19f18b39ebb985b42f852634a46116601f4dcae7133450
Sep 05 02:17:49.216879 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.216835    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd978685_2990_4515_8493_ae261615757d.slice/crio-0fc616d910aac4f3e87177cb2b91ebcd093dbfd131699bea972a96eb5f954771 WatchSource:0}: Error finding container 0fc616d910aac4f3e87177cb2b91ebcd093dbfd131699bea972a96eb5f954771: Status 404 returned error can't find the container with id 0fc616d910aac4f3e87177cb2b91ebcd093dbfd131699bea972a96eb5f954771
Sep 05 02:17:49.217927 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.217782    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552d8eb0_1a34_4f95_af5f_3c863be9df7f.slice/crio-02b8a4258e95fda481f87047caeeb3e242fb889f7ef36b290f87894280c358a0 WatchSource:0}: Error finding container 02b8a4258e95fda481f87047caeeb3e242fb889f7ef36b290f87894280c358a0: Status 404 returned error can't find the container with id 02b8a4258e95fda481f87047caeeb3e242fb889f7ef36b290f87894280c358a0
Sep 05 02:17:49.219821 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.219804    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda657a2f5_bf54_4195_a98b_dd5e594ad5e7.slice/crio-620135009fcd778d9a8d80ad644b6eba939ac03a604f5fac87a06679329154bb WatchSource:0}: Error finding container 620135009fcd778d9a8d80ad644b6eba939ac03a604f5fac87a06679329154bb: Status 404 returned error can't find the container with id 620135009fcd778d9a8d80ad644b6eba939ac03a604f5fac87a06679329154bb
Sep 05 02:17:49.220785 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.220589    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f3509_edfb_416a_b956_b469a9841b91.slice/crio-2091edf9a5d8d7a7d88915f50318f71ebb15f9afbae2f84902fde9f9c717207e WatchSource:0}: Error finding container 2091edf9a5d8d7a7d88915f50318f71ebb15f9afbae2f84902fde9f9c717207e: Status 404 returned error can't find the container with id 2091edf9a5d8d7a7d88915f50318f71ebb15f9afbae2f84902fde9f9c717207e
Sep 05 02:17:49.222370 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.222229    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8efa68_e2e0_478d_8244_a1433ec4c1a2.slice/crio-b28493220d6d2a7195b3c92d204214f30c16da5b1b4e4ab9eaa35f71f79ece4d WatchSource:0}: Error finding container b28493220d6d2a7195b3c92d204214f30c16da5b1b4e4ab9eaa35f71f79ece4d: Status 404 returned error can't find the container with id b28493220d6d2a7195b3c92d204214f30c16da5b1b4e4ab9eaa35f71f79ece4d
Sep 05 02:17:49.223662 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:17:49.223238    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629c6726_442d_4ee3_99a5_323e477b1e53.slice/crio-b97d10729ed6916c7233cf269c5b4feaaa603eee3c1b9dcebb27ddef955794a7 WatchSource:0}: Error finding container b97d10729ed6916c7233cf269c5b4feaaa603eee3c1b9dcebb27ddef955794a7: Status 404 returned error can't find the container with id b97d10729ed6916c7233cf269c5b4feaaa603eee3c1b9dcebb27ddef955794a7
Sep 05 02:17:49.738456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:49.738178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:49.738901 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:49.738531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:49.738901 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.738674    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:49.738901 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.738738    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.738719637 +0000 UTC m=+6.060492141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:49.739246 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.739203    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:49.739313 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.739267    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.739249577 +0000 UTC m=+6.061022079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:49.940504 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:49.940462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:49.940672 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.940648    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:49.940672 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.940673    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:49.940839 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.940686    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:49.940839 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:49.940749    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.940730342 +0000 UTC m=+6.262502842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:50.157408 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.157043    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:48 +0000 UTC" deadline="2028-04-01 10:46:09.062937485 +0000 UTC"
Sep 05 02:17:50.157408 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.157088    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13784h28m18.905855198s"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.180473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:50.180603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.181034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:50.181162    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.181226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:50.181369 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:50.181292    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:17:50.206027 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.205243    2582 generic.go:356] "Generic (PLEG): container finished" podID="59ee296aad7c4e56d27fc66bd1b97d18" containerID="422725cb1d6d5ecf2ed0c26e034ed103dac78d7d72c1a659a5e667400a2d02d5" exitCode=0
Sep 05 02:17:50.206027 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.205388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerDied","Data":"422725cb1d6d5ecf2ed0c26e034ed103dac78d7d72c1a659a5e667400a2d02d5"}
Sep 05 02:17:50.221358 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.221318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" event={"ID":"58007dd99525496ea22ff71e4a80cd63","Type":"ContainerStarted","Data":"664aa2aaecf5ef8ec9d8e78b00a1d05af316d5163278c406c7b28a57cf646dde"}
Sep 05 02:17:50.235259 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.235182    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-142.ec2.internal" podStartSLOduration=3.235158405 podStartE2EDuration="3.235158405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:50.235091326 +0000 UTC m=+4.556863847" watchObservedRunningTime="2026-09-05 02:17:50.235158405 +0000 UTC m=+4.556930926"
Sep 05 02:17:50.240699 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.240665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bh2bf" event={"ID":"629c6726-442d-4ee3-99a5-323e477b1e53","Type":"ContainerStarted","Data":"b97d10729ed6916c7233cf269c5b4feaaa603eee3c1b9dcebb27ddef955794a7"}
Sep 05 02:17:50.246836 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.246787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjfmw" event={"ID":"bd978685-2990-4515-8493-ae261615757d","Type":"ContainerStarted","Data":"0fc616d910aac4f3e87177cb2b91ebcd093dbfd131699bea972a96eb5f954771"}
Sep 05 02:17:50.258707 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.258671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lclfz" event={"ID":"0b1f3509-edfb-416a-b956-b469a9841b91","Type":"ContainerStarted","Data":"2091edf9a5d8d7a7d88915f50318f71ebb15f9afbae2f84902fde9f9c717207e"}
Sep 05 02:17:50.261353 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.261324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm" event={"ID":"a657a2f5-bf54-4195-a98b-dd5e594ad5e7","Type":"ContainerStarted","Data":"620135009fcd778d9a8d80ad644b6eba939ac03a604f5fac87a06679329154bb"}
Sep 05 02:17:50.275670 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.275629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"02b8a4258e95fda481f87047caeeb3e242fb889f7ef36b290f87894280c358a0"}
Sep 05 02:17:50.291857 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.291801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kfh4z" event={"ID":"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7","Type":"ContainerStarted","Data":"febe5b190446c8d5fa19f18b39ebb985b42f852634a46116601f4dcae7133450"}
Sep 05 02:17:50.298893 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.296029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerStarted","Data":"b28493220d6d2a7195b3c92d204214f30c16da5b1b4e4ab9eaa35f71f79ece4d"}
Sep 05 02:17:50.308176 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.308139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cplkx" event={"ID":"84933ea4-b102-42f9-b2e9-5f47c655130c","Type":"ContainerStarted","Data":"a77a0ea6a805f33ab553bccaea72297749fd7e94e1079ef0bb89295328b32910"}
Sep 05 02:17:50.319090 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:50.319054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw72b" event={"ID":"d06cc8bd-5304-4d9b-8737-48a1f5370e22","Type":"ContainerStarted","Data":"d3690c824b9ca0bfc3a1f5ad1b8cd8e4dc7272851708eb0df1cf368c167fa70c"}
Sep 05 02:17:51.346406 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:51.346367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" event={"ID":"59ee296aad7c4e56d27fc66bd1b97d18","Type":"ContainerStarted","Data":"a4e69a91766ce7522c2e6169528becfb086047068bac2fb8511b525c2c5fb034"}
Sep 05 02:17:51.758049 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:51.757960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:51.758049 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:51.758024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:51.758251 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.758170    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.758251 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.758233    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:55.758215318 +0000 UTC m=+10.079987837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.758663 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.758641    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.758836 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.758696    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:55.758681501 +0000 UTC m=+10.080454003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.960732 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:51.960665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:51.960921 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.960888    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:51.960921 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.960908    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:51.960921 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.960920    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:51.961095 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:51.960981    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:55.960962289 +0000 UTC m=+10.282734803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:52.181041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:52.180959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:52.181205 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:52.181089    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:52.181205 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:52.181163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:52.181310 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:52.181247    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:52.181310 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:52.181300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:52.181408 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:52.181362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:17:54.180211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:54.180180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:54.180662 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:54.180307    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:54.180741 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:54.180715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:54.180875 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:54.180848    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:54.180974 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:54.180903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:54.181030 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:54.180975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:17:55.796774 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:55.796737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:55.797225 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:55.796805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:55.797225 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.796956    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:55.797225 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.796962    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:55.797225 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.797020    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.797003012 +0000 UTC m=+18.118775508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:55.797225 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.797039    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.797030867 +0000 UTC m=+18.118803363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:55.998878 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:55.998259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:55.998878 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.998419    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:55.998878 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.998437    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:55.998878 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.998449    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:55.998878 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:55.998506    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.998489374 +0000 UTC m=+18.320261876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:56.183296 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:56.183224    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:56.183436 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:56.183227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:56.183436 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:56.183352    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:17:56.183436 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:56.183399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:56.183537 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:56.183481    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:56.183591 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:56.183572    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:58.178966 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:58.178744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:17:58.178966 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:58.178892    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:17:58.178966 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:58.178749    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:17:58.178966 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:17:58.178751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:17:58.179468 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:58.178997    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:17:58.179468 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:17:58.179071    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:00.178635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:00.178599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:00.178635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:00.178599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:00.178635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:00.178630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:00.179131 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:00.178739    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:00.179131 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:00.178830    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:00.179131 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:00.178910    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:02.181052 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:02.181025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:02.181462 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:02.181025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:02.181462 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:02.181133    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:02.181462 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:02.181026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:02.181462 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:02.181235    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:02.181462 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:02.181342    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:03.860265 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:03.860223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:03.860713 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:03.860278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:03.860713 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:03.860389    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:03.860713 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:03.860390    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:03.860713 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:03.860459    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.860437197 +0000 UTC m=+34.182209724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:03.860713 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:03.860481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.860470422 +0000 UTC m=+34.182242918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:04.061730 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:04.061687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:04.061930 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.061861    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:04.061930 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.061888    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:04.061930 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.061902    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:04.062076 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.061973    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.061951882 +0000 UTC m=+34.383724380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:04.178943 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:04.178867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:04.179110 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:04.178867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:04.179110 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.179017    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:04.179110 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:04.178868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:04.179274 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.179111    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:04.179274 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:04.179163    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:06.178886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:06.178851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:06.179373 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:06.178958    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:06.179373 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:06.179051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:06.179373 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:06.179177    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:06.179373 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:06.179234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:06.179373 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:06.179306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:08.178576 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.178354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:08.179366 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.178365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:08.179366 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.178374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:08.179366 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:08.178726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:08.179366 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:08.178597    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:08.179366 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:08.178804    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:08.384880 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.384844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bh2bf" event={"ID":"629c6726-442d-4ee3-99a5-323e477b1e53","Type":"ContainerStarted","Data":"97a303349541aaf75b4c5af2e63d67722621237758984032b622145bbd082d4a"}
Sep 05 02:18:08.386383 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.386357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lclfz" event={"ID":"0b1f3509-edfb-416a-b956-b469a9841b91","Type":"ContainerStarted","Data":"6896fd80f4a6f9f68da94853b774d4a8817d1a7efec98878291eec3e6b0e4279"}
Sep 05 02:18:08.387847 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.387821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm" event={"ID":"a657a2f5-bf54-4195-a98b-dd5e594ad5e7","Type":"ContainerStarted","Data":"a5cc6741baeb16a35a92bc7c802ba451f967bf84fab6daee52d27936102f0cb7"}
Sep 05 02:18:08.390491 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.390472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:18:08.390856 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.390829    2582 generic.go:356] "Generic (PLEG): container finished" podID="552d8eb0-1a34-4f95-af5f-3c863be9df7f" containerID="24abe9f2698a9fbb901a38ae90ef82ae3259f7211739c0fe2b4dd336e51627b1" exitCode=1
Sep 05 02:18:08.390937 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.390873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"80f1d7dd0a36948243da4faa0eca218da86d0cb682360410180402eabe77b8b5"}
Sep 05 02:18:08.390937 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.390931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"0e65d8bb8d7a225958d9534d7b1bbffc490555cd310484b0f1b18fe0c6c1e099"}
Sep 05 02:18:08.391073 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.390974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"a684bdee85fd5cbd982088aba19bed9ac68f01d080eae9a7ecd212c4ad7f50cb"}
Sep 05 02:18:08.391073 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.391017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"029da5e47dfde4ad87908b1b021523e4ff6a072fb39bb5c7d01f4bd65215d7e7"}
Sep 05 02:18:08.391073 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.391054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerDied","Data":"24abe9f2698a9fbb901a38ae90ef82ae3259f7211739c0fe2b4dd336e51627b1"}
Sep 05 02:18:08.391192 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.391084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"4ada22b6433e1ff737f8b91932dad3740d32601daf7e97f3dfe17d2cbdc30124"}
Sep 05 02:18:08.392396 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.392375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kfh4z" event={"ID":"18ca5a4f-3782-4a35-9e89-2c34cc0f96b7","Type":"ContainerStarted","Data":"88bca566f4f72e50b66630420efa03ec45c6da3ef512b20b0a47168a46ab86d4"}
Sep 05 02:18:08.393781 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.393737    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="36dc75775beb41d3ec1ac5ce904ceef849670eed536034a43b5fdbdfc56f1046" exitCode=0
Sep 05 02:18:08.393855 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.393819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"36dc75775beb41d3ec1ac5ce904ceef849670eed536034a43b5fdbdfc56f1046"}
Sep 05 02:18:08.395369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.395347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cplkx" event={"ID":"84933ea4-b102-42f9-b2e9-5f47c655130c","Type":"ContainerStarted","Data":"4498a6bebc5bb343d793d72cd687880af1502292683e02f40bcf28039a725046"}
Sep 05 02:18:08.396881 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.396845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw72b" event={"ID":"d06cc8bd-5304-4d9b-8737-48a1f5370e22","Type":"ContainerStarted","Data":"4dea3cf788e16830e25408ebe15193545119bfb862287d62a3b8773f4787715d"}
Sep 05 02:18:08.404485 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.404436    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bh2bf" podStartSLOduration=4.557096747 podStartE2EDuration="22.404414625s" totalImagesPullingTime="17.847317878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:08.404065049 +0000 UTC m=+22.725837566" watchObservedRunningTime="2026-09-05 02:18:08.404414625 +0000 UTC m=+22.726187143"
Sep 05 02:18:08.404834 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.404807    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-142.ec2.internal" podStartSLOduration=21.404794549000002 podStartE2EDuration="21.404794549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:51.36358887 +0000 UTC m=+5.685361397" watchObservedRunningTime="2026-09-05 02:18:08.404794549 +0000 UTC m=+22.726567067"
Sep 05 02:18:08.418321 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.418287    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lclfz" podStartSLOduration=4.564047768 podStartE2EDuration="22.418277461s" totalImagesPullingTime="17.854229693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:08.418063361 +0000 UTC m=+22.739835941" watchObservedRunningTime="2026-09-05 02:18:08.418277461 +0000 UTC m=+22.740049978"
Sep 05 02:18:08.432914 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.432859    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cplkx" podStartSLOduration=4.639392148 podStartE2EDuration="22.432847984s" totalImagesPullingTime="17.793455836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:08.432615908 +0000 UTC m=+22.754388425" watchObservedRunningTime="2026-09-05 02:18:08.432847984 +0000 UTC m=+22.754620507"
Sep 05 02:18:08.465455 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.465412    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kfh4z" podStartSLOduration=4.6120335489999995 podStartE2EDuration="22.465400072s" totalImagesPullingTime="17.853366523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:08.448566474 +0000 UTC m=+22.770338992" watchObservedRunningTime="2026-09-05 02:18:08.465400072 +0000 UTC m=+22.787172589"
Sep 05 02:18:08.465584 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.465566    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bw72b" podStartSLOduration=4.19916398 podStartE2EDuration="22.465558813s" totalImagesPullingTime="18.266394833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:08.465070344 +0000 UTC m=+22.786842872" watchObservedRunningTime="2026-09-05 02:18:08.465558813 +0000 UTC m=+22.787331330"
Sep 05 02:18:08.789401 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:08.789375    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:09.189489 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.189325    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:08.789394539Z","UUID":"b25ffca9-fd54-4e87-8a44-73b4068b6910","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:09.192261 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.192241    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:09.192372 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.192271    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:09.401824 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.401784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gjfmw" event={"ID":"bd978685-2990-4515-8493-ae261615757d","Type":"ContainerStarted","Data":"17edc4863e21ad7ae65bebe0f8289181b6dc8d5e8006c2ffe0ef642022359b32"}
Sep 05 02:18:09.404846 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.404798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm" event={"ID":"a657a2f5-bf54-4195-a98b-dd5e594ad5e7","Type":"ContainerStarted","Data":"d2220f644ed4f64241064a57b5100cfcaa48c342a0b4d6a782cf241500e37e9e"}
Sep 05 02:18:09.416731 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:09.416687    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gjfmw" podStartSLOduration=5.563310106 podStartE2EDuration="23.416671288s" totalImagesPullingTime="17.853361182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:09.41629481 +0000 UTC m=+23.738067344" watchObservedRunningTime="2026-09-05 02:18:09.416671288 +0000 UTC m=+23.738443805"
Sep 05 02:18:10.024109 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.024081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:10.145913 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.145881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:10.178379 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.178339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:10.178557 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.178339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:10.178557 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:10.178479    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:10.178557 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.178339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:10.178557 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:10.178549    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:10.178793 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:10.178594    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:10.409587 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.409487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm" event={"ID":"a657a2f5-bf54-4195-a98b-dd5e594ad5e7","Type":"ContainerStarted","Data":"8f1da02210c87202643bcbb53ee7b3107aa4eaf129d9515f9933ca26a838e56e"}
Sep 05 02:18:10.413122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.413096    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:18:10.413711 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.413681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"f8af4037ed22481844f02e97d26ff08053a28495b3967add8d595813a74d3fbb"}
Sep 05 02:18:10.429274 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:10.429223    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cxqm" podStartSLOduration=4.379961477 podStartE2EDuration="24.42920891s" totalImagesPullingTime="20.049247433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.428624983 +0000 UTC m=+24.750397522" watchObservedRunningTime="2026-09-05 02:18:10.42920891 +0000 UTC m=+24.750981428"
Sep 05 02:18:11.868506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:11.868284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:12.178331 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:12.178247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:12.178488 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:12.178247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:12.178488 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:12.178423    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:12.178488 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:12.178454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:12.178661 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:12.178593    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:12.178716 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:12.178687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:12.209043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:12.209011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:18:12.209753 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:12.209729    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:18:13.424884 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.424707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:18:13.425609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.425257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"16a508f9f14516331cbf1276c8fa068c70c0776ca171800f558071d1245d1fe1"}
Sep 05 02:18:13.425609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.425523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:13.425749 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.425729    2582 scope.go:117] "RemoveContainer" containerID="24abe9f2698a9fbb901a38ae90ef82ae3259f7211739c0fe2b4dd336e51627b1"
Sep 05 02:18:13.427355 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.427335    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="c6f3808bd3b60054001158aac04ae3d4e5248d27b9c3edeb5a5bdd680b23bebd" exitCode=0
Sep 05 02:18:13.427466 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.427367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"c6f3808bd3b60054001158aac04ae3d4e5248d27b9c3edeb5a5bdd680b23bebd"}
Sep 05 02:18:13.443373 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.443351    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:13.481854 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.481541    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:18:13.481854 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.481664    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 02:18:13.482524 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.482500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cplkx"
Sep 05 02:18:13.600164 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:13.600133    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:14.178908 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.178881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:14.179043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.178880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:14.179043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.179025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:14.179162 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.179022    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:14.179226 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.179170    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:14.179296 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.179269    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:14.433575 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.433499    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:18:14.433978 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.433891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" event={"ID":"552d8eb0-1a34-4f95-af5f-3c863be9df7f","Type":"ContainerStarted","Data":"4847cd0ccc4c727322640b899b3bbbfae736e22397f388dfe1184c962c767f5f"}
Sep 05 02:18:14.434109 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.434089    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:14.434170 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.434162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:14.436214 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.436189    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="cb9a2b0fc7bfc74c242c77639fa5b349d7bbe0eb8875f66e4e84e4c21a2040d6" exitCode=0
Sep 05 02:18:14.436325 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.436218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"cb9a2b0fc7bfc74c242c77639fa5b349d7bbe0eb8875f66e4e84e4c21a2040d6"}
Sep 05 02:18:14.452268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.452247    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:14.461198 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.461174    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxl5g"]
Sep 05 02:18:14.461302 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.461256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:14.461353 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.461332    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:14.471898 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.471871    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8dhc"]
Sep 05 02:18:14.472021 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.472008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:14.472113 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.472092    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:14.473334 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.473303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2nhpg"]
Sep 05 02:18:14.473424 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.473398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:14.473500 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:14.473482    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:14.523212 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:14.523153    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9" podStartSLOduration=10.4654575 podStartE2EDuration="28.523136676s" totalImagesPullingTime="18.057679176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:14.4974556 +0000 UTC m=+28.819228167" watchObservedRunningTime="2026-09-05 02:18:14.523136676 +0000 UTC m=+28.844909193"
Sep 05 02:18:15.441268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:15.441081    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="690ef263ed27f1aeea5ee0d18e3731c5f7f780f1c2d6ff568879566d5c31c4cf" exitCode=0
Sep 05 02:18:15.441676 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:15.441162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"690ef263ed27f1aeea5ee0d18e3731c5f7f780f1c2d6ff568879566d5c31c4cf"}
Sep 05 02:18:15.522556 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:15.522524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:15.620682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:15.620651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:16.184778 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:16.184733    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:16.184938 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:16.184879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:16.185290 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:16.185270    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:16.185397 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:16.185379    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:16.185453 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:16.185441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:16.185525 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:16.185508    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:16.497732 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:16.497644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:16.559733 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:16.559699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:18.178682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:18.178647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:18.179515 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:18.178839    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:18.179515 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:18.178851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:18.179515 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:18.178875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:18.179515 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:18.178960    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:18.179515 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:18.179073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:19.870537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:19.870490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:19.870537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:19.870547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:19.871160 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:19.870644    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:19.871160 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:19.870679    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:19.871160 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:19.870715    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret podName:79bd5ce1-cb17-4035-984a-dfc16c53b637 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:51.870700198 +0000 UTC m=+66.192472695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret") pod "global-pull-secret-syncer-w8dhc" (UID: "79bd5ce1-cb17-4035-984a-dfc16c53b637") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:19.871160 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:19.870739    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:51.870722572 +0000 UTC m=+66.192495067 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:20.072524 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.072472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:20.072709 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.072609    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:20.072709 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.072624    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:20.072709 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.072632    2582 projected.go:196] Error preparing data for projected volume kube-api-access-n4n9s for pod openshift-network-diagnostics/network-check-target-xxl5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.072709 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.072683    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s podName:45394a99-f247-4de3-aabb-ce4fd06ed5f0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.07267179 +0000 UTC m=+66.394444285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n4n9s" (UniqueName: "kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s") pod "network-check-target-xxl5g" (UID: "45394a99-f247-4de3-aabb-ce4fd06ed5f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.181301 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.181267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:20.181466 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.181355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:20.181466 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.181362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxl5g" podUID="45394a99-f247-4de3-aabb-ce4fd06ed5f0"
Sep 05 02:18:20.181466 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.181387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:20.181570 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.181467    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dhc" podUID="79bd5ce1-cb17-4035-984a-dfc16c53b637"
Sep 05 02:18:20.181570 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.181533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2nhpg" podUID="37c79c34-5580-45bb-85b2-b82afd3dd6e2"
Sep 05 02:18:20.457093 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.456995    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="2ea82df59e779473692631cb371de6a28d896de860ec5c18a4cd50c5032bd53f" exitCode=0
Sep 05 02:18:20.457093 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.457044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"2ea82df59e779473692631cb371de6a28d896de860ec5c18a4cd50c5032bd53f"}
Sep 05 02:18:20.498799 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.498734    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-142.ec2.internal" event="NodeReady"
Sep 05 02:18:20.498921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.498891    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:20.547459 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.547430    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7p6tl"]
Sep 05 02:18:20.572453 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.572426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:20.575997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.575858    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m8tn8"]
Sep 05 02:18:20.575997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.575956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.578402 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.578380    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:20.578517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.578413    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:20.578699 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.578686    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l69pm\""
Sep 05 02:18:20.592111 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.592082    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7p6tl"]
Sep 05 02:18:20.592231 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.592124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.592231 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.592143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8tn8"]
Sep 05 02:18:20.596306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.596280    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:20.596306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.596302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:20.596456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.596287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2vlql\""
Sep 05 02:18:20.596456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.596330    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:20.676615 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdvd\" (UniqueName: \"kubernetes.io/projected/206465c0-c8d6-4238-b7aa-14409ee968f1-kube-api-access-zjdvd\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.676615 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.676821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33c51d59-1ee4-456c-acdf-6f4471457da0-config-volume\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.676821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c51d59-1ee4-456c-acdf-6f4471457da0-tmp-dir\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.676821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206465c0-c8d6-4238-b7aa-14409ee968f1-cert\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.676821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.676779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2z4\" (UniqueName: \"kubernetes.io/projected/33c51d59-1ee4-456c-acdf-6f4471457da0-kube-api-access-tt2z4\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.708169 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.708100    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:20.777337 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdvd\" (UniqueName: \"kubernetes.io/projected/206465c0-c8d6-4238-b7aa-14409ee968f1-kube-api-access-zjdvd\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.777337 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33c51d59-1ee4-456c-acdf-6f4471457da0-config-volume\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c51d59-1ee4-456c-acdf-6f4471457da0-tmp-dir\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206465c0-c8d6-4238-b7aa-14409ee968f1-cert\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.777485    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:20.777558    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:21.277538765 +0000 UTC m=+35.599311280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:20.777590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2z4\" (UniqueName: \"kubernetes.io/projected/33c51d59-1ee4-456c-acdf-6f4471457da0-kube-api-access-tt2z4\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.777921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.777865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c51d59-1ee4-456c-acdf-6f4471457da0-tmp-dir\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.778156 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.778131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33c51d59-1ee4-456c-acdf-6f4471457da0-config-volume\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.778529 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.778508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:20.781847 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.781829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206465c0-c8d6-4238-b7aa-14409ee968f1-cert\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.787852 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.787827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2z4\" (UniqueName: \"kubernetes.io/projected/33c51d59-1ee4-456c-acdf-6f4471457da0-kube-api-access-tt2z4\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:20.789162 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.789141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdvd\" (UniqueName: \"kubernetes.io/projected/206465c0-c8d6-4238-b7aa-14409ee968f1-kube-api-access-zjdvd\") pod \"ingress-canary-m8tn8\" (UID: \"206465c0-c8d6-4238-b7aa-14409ee968f1\") " pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:20.901189 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:20.901149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8tn8"
Sep 05 02:18:21.208240 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:21.208021    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8tn8"]
Sep 05 02:18:21.213697 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:18:21.213660    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206465c0_c8d6_4238_b7aa_14409ee968f1.slice/crio-a920a75c4796eb5ac370cb0fd99467b54872e9fb0e0d447b647e14d842d1823f WatchSource:0}: Error finding container a920a75c4796eb5ac370cb0fd99467b54872e9fb0e0d447b647e14d842d1823f: Status 404 returned error can't find the container with id a920a75c4796eb5ac370cb0fd99467b54872e9fb0e0d447b647e14d842d1823f
Sep 05 02:18:21.280348 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:21.280263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:21.280494 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:21.280415    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:21.280494 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:21.280483    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.280467993 +0000 UTC m=+36.602240489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:21.460318 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:21.460282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8tn8" event={"ID":"206465c0-c8d6-4238-b7aa-14409ee968f1","Type":"ContainerStarted","Data":"a920a75c4796eb5ac370cb0fd99467b54872e9fb0e0d447b647e14d842d1823f"}
Sep 05 02:18:21.463330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:21.463305    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf8efa68-e2e0-478d-8244-a1433ec4c1a2" containerID="abc39d770dd17bcc761afe8392071b3ca2867550323d77ead55a399d3ebc042a" exitCode=0
Sep 05 02:18:21.463460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:21.463342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerDied","Data":"abc39d770dd17bcc761afe8392071b3ca2867550323d77ead55a399d3ebc042a"}
Sep 05 02:18:22.181906 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.181857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:22.181906 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.181893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:22.182515 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.182108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:22.184510 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.184485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.184667 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.184559    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:22.185291 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.185271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hwbs\""
Sep 05 02:18:22.185504 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.185487    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.185681 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.185667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:22.185917 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.185898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66tlm\""
Sep 05 02:18:22.289437 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.289399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:22.289621 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:22.289529    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:22.289621 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:22.289604    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:24.289586597 +0000 UTC m=+38.611359113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:22.469921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.469827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpws9" event={"ID":"bf8efa68-e2e0-478d-8244-a1433ec4c1a2","Type":"ContainerStarted","Data":"c62aa85cc74ea4b321d8c299884b72b61e442aadfa6f4643ae0d73d692f17db3"}
Sep 05 02:18:22.492159 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.492097    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bpws9" podStartSLOduration=8.661413214 podStartE2EDuration="36.492077829s" totalImagesPullingTime="27.830664615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:22.490558398 +0000 UTC m=+36.812330956" watchObservedRunningTime="2026-09-05 02:18:22.492077829 +0000 UTC m=+36.813850346"
Sep 05 02:18:22.610483 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.610448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:22.672433 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:22.672397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.378985 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:23.378947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.445148 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:23.445120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:24.304788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:24.304600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:24.304788 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:24.304781    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:24.304998 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:24.304852    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:28.304836498 +0000 UTC m=+42.626608994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:24.475227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:24.475194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8tn8" event={"ID":"206465c0-c8d6-4238-b7aa-14409ee968f1","Type":"ContainerStarted","Data":"9f6d289d727b82853b4676541aa65d06db14fd79ed35ecc94d3f02b90b7403b1"}
Sep 05 02:18:24.491238 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:24.491139    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m8tn8" podStartSLOduration=2.190345575 podStartE2EDuration="4.49112302s" totalImagesPullingTime="2.300777445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:24.49072175 +0000 UTC m=+38.812494279" watchObservedRunningTime="2026-09-05 02:18:24.49112302 +0000 UTC m=+38.812895532"
Sep 05 02:18:25.663810 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:25.663778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:25.763799 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:25.763746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:27.365207 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:27.365172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:27.426687 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:27.426653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:28.333286 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:28.333239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:28.333472 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:28.333411    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:28.333513 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:28.333481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:36.33346423 +0000 UTC m=+50.655236725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:29.571416 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:29.571382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:29.661079 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:29.661050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:32.634837 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:32.634801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:32.721619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:32.721589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:35.867966 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:35.867928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:35.927368 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:35.927333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:36.390123 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:36.390090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:36.390274 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:36.390255    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:36.390343 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:36.390333    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.390317033 +0000 UTC m=+66.712089529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:44.839876 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.839835    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"]
Sep 05 02:18:44.842251 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.842223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:44.844559 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.844528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 05 02:18:44.844559 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.844550    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cf2dm\""
Sep 05 02:18:44.844737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.844594    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.845521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.845499    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 05 02:18:44.845614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.845554    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.853578 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.853556    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"]
Sep 05 02:18:44.949404 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.949368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sc5v\" (UniqueName: \"kubernetes.io/projected/4f7ef39f-8036-4d72-9221-fb17307f199d-kube-api-access-8sc5v\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:44.949404 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.949401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7ef39f-8036-4d72-9221-fb17307f199d-serving-cert\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:44.949622 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.949421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f7ef39f-8036-4d72-9221-fb17307f199d-config\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:44.959047 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.959013    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c954d494-7nrt6"]
Sep 05 02:18:44.960726 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.960705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:44.963460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963438    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 05 02:18:44.963647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963630    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 05 02:18:44.963737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963680    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 05 02:18:44.963861 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963844    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.963934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.963992 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.963946    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 05 02:18:44.964043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.964028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j86s6\""
Sep 05 02:18:44.977720 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:44.977696    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c954d494-7nrt6"]
Sep 05 02:18:45.049869 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.049837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sc5v\" (UniqueName: \"kubernetes.io/projected/4f7ef39f-8036-4d72-9221-fb17307f199d-kube-api-access-8sc5v\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.050051 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.049878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7ef39f-8036-4d72-9221-fb17307f199d-serving-cert\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.050051 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.049896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f7ef39f-8036-4d72-9221-fb17307f199d-config\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.050399 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.050381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f7ef39f-8036-4d72-9221-fb17307f199d-config\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.053885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.053861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7ef39f-8036-4d72-9221-fb17307f199d-serving-cert\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.060149 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.060125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sc5v\" (UniqueName: \"kubernetes.io/projected/4f7ef39f-8036-4d72-9221-fb17307f199d-kube-api-access-8sc5v\") pod \"service-ca-operator-859f94c4b-487ng\" (UID: \"4f7ef39f-8036-4d72-9221-fb17307f199d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.128623 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.128581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:45.150724 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-stats-auth\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.150845 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-default-certificate\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.150845 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.150845 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.150845 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcd8r\" (UniqueName: \"kubernetes.io/projected/131bce09-5bd6-4376-9aff-440919804a4d-kube-api-access-mcd8r\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.150965 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.150843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"
Sep 05 02:18:45.223152 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.223122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:45.252275 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.252243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-stats-auth\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.252395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.252288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-default-certificate\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.252395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.252315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.252395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.252341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.252395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.252365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcd8r\" (UniqueName: \"kubernetes.io/projected/131bce09-5bd6-4376-9aff-440919804a4d-kube-api-access-mcd8r\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.252601 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.252485    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:45.252601 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.252513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.752488861 +0000 UTC m=+60.074261357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:45.252601 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.252560    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.752542051 +0000 UTC m=+60.074314562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : secret "router-metrics-certs-default" not found
Sep 05 02:18:45.254842 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.254815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-default-certificate\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.254962 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.254843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-stats-auth\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.261436 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.261411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcd8r\" (UniqueName: \"kubernetes.io/projected/131bce09-5bd6-4376-9aff-440919804a4d-kube-api-access-mcd8r\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.419787 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.419682    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng"]
Sep 05 02:18:45.422687 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:18:45.422661    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7ef39f_8036_4d72_9221_fb17307f199d.slice/crio-2968e1ddd0d10d44667bf006c068657038e5d145ed8508ba7f218ff225892dcf WatchSource:0}: Error finding container 2968e1ddd0d10d44667bf006c068657038e5d145ed8508ba7f218ff225892dcf: Status 404 returned error can't find the container with id 2968e1ddd0d10d44667bf006c068657038e5d145ed8508ba7f218ff225892dcf
Sep 05 02:18:45.529600 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.529556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng" event={"ID":"4f7ef39f-8036-4d72-9221-fb17307f199d","Type":"ContainerStarted","Data":"2968e1ddd0d10d44667bf006c068657038e5d145ed8508ba7f218ff225892dcf"}
Sep 05 02:18:45.757191 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.757082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.757191 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:45.757145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:45.757387 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.757290    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.757272064 +0000 UTC m=+61.079044581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:45.757387 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.757305    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:45.757387 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:45.757355    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.757343148 +0000 UTC m=+61.079115648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : secret "router-metrics-certs-default" not found
Sep 05 02:18:46.456464 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:46.456432    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b72d9"
Sep 05 02:18:46.627706 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:46.627660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:46.728697 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:46.728584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:46.766631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:46.766591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:46.766836 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:46.766653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:46.766836 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:46.766718    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.766697673 +0000 UTC m=+63.088470173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:46.766836 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:46.766785    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:46.766836 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:46.766832    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.76682263 +0000 UTC m=+63.088595126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : secret "router-metrics-certs-default" not found
Sep 05 02:18:47.535593 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:47.535553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng" event={"ID":"4f7ef39f-8036-4d72-9221-fb17307f199d","Type":"ContainerStarted","Data":"99e952954e90dd32e8fa60c01962e2a9e5c4f8d15be3ae5f390f48f5d9f5cd6d"}
Sep 05 02:18:47.568419 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:47.568367    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng" podStartSLOduration=1.613408867 podStartE2EDuration="3.568350381s" totalImagesPullingTime="1.954941514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:47.568005449 +0000 UTC m=+61.889777977" watchObservedRunningTime="2026-09-05 02:18:47.568350381 +0000 UTC m=+61.890122899"
Sep 05 02:18:47.572316 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:47.572293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:47.645499 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:47.645406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:48.778897 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:48.778852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:48.779320 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:48.778933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:48.779320 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:48.779035    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.779010403 +0000 UTC m=+67.100782914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:48.779320 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:48.779072    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:48.779320 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:48.779130    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.779116719 +0000 UTC m=+67.100889218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : secret "router-metrics-certs-default" not found
Sep 05 02:18:48.918307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:48.918270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:48.984530 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:48.984497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.376139 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:49.376102    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.631438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:49.631364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.733490 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:49.733459    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.328548 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.328514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.386719 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.386685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.494049 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.494010    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c6f75"]
Sep 05 02:18:51.515927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.515899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.528576 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.528555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6f75"]
Sep 05 02:18:51.528685 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.528665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.531998 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.531977    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:18:51.532294 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.532278    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:18:51.532391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.532370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwbbc\""
Sep 05 02:18:51.532476 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.532374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:18:51.532967 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.532953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:18:51.554984 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.554960    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"]
Sep 05 02:18:51.578807 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.578723    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"]
Sep 05 02:18:51.578920 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.578833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"
Sep 05 02:18:51.581680 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.581660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 05 02:18:51.581819 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.581668    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:51.582084 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.582070    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4fdzl\""
Sep 05 02:18:51.597996 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.597975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc698016-9978-4b50-9bb0-b80f78de1181-data-volume\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.598120 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.598120 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc698016-9978-4b50-9bb0-b80f78de1181-crio-socket\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.598216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpfv\" (UniqueName: \"kubernetes.io/projected/bc698016-9978-4b50-9bb0-b80f78de1181-kube-api-access-xhpfv\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.598216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxmh\" (UniqueName: \"kubernetes.io/projected/3a5650fe-be9b-4c6f-8428-8193261874ad-kube-api-access-hxxmh\") pod \"migrator-fdbb8fdff-hw5qt\" (UID: \"3a5650fe-be9b-4c6f-8428-8193261874ad\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"
Sep 05 02:18:51.598216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.598886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.598869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.698535 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc698016-9978-4b50-9bb0-b80f78de1181-crio-socket\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpfv\" (UniqueName: \"kubernetes.io/projected/bc698016-9978-4b50-9bb0-b80f78de1181-kube-api-access-xhpfv\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.698584    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxmh\" (UniqueName: \"kubernetes.io/projected/3a5650fe-be9b-4c6f-8428-8193261874ad-kube-api-access-hxxmh\") pod \"migrator-fdbb8fdff-hw5qt\" (UID: \"3a5650fe-be9b-4c6f-8428-8193261874ad\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc698016-9978-4b50-9bb0-b80f78de1181-crio-socket\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.698693    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.198676016 +0000 UTC m=+66.520448514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:51.698815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.699211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc698016-9978-4b50-9bb0-b80f78de1181-data-volume\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.699211 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.698844    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:51.699211 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.698904    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.198887805 +0000 UTC m=+66.520660301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:51.699211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.698954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:51.699211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.699137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc698016-9978-4b50-9bb0-b80f78de1181-data-volume\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.710088 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.710057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxmh\" (UniqueName: \"kubernetes.io/projected/3a5650fe-be9b-4c6f-8428-8193261874ad-kube-api-access-hxxmh\") pod \"migrator-fdbb8fdff-hw5qt\" (UID: \"3a5650fe-be9b-4c6f-8428-8193261874ad\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"
Sep 05 02:18:51.712577 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.712559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpfv\" (UniqueName: \"kubernetes.io/projected/bc698016-9978-4b50-9bb0-b80f78de1181-kube-api-access-xhpfv\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:51.887351 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.887317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"
Sep 05 02:18:51.899900 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.899878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:18:51.900029 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.899959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:51.903273 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.903253    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:51.903273 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.903266    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:51.910268 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.910246    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:18:51.910350 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:51.910317    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs podName:37c79c34-5580-45bb-85b2-b82afd3dd6e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:55.91029554 +0000 UTC m=+130.232068053 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs") pod "network-metrics-daemon-2nhpg" (UID: "37c79c34-5580-45bb-85b2-b82afd3dd6e2") : secret "metrics-daemon-secret" not found
Sep 05 02:18:51.912741 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:51.912717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79bd5ce1-cb17-4035-984a-dfc16c53b637-original-pull-secret\") pod \"global-pull-secret-syncer-w8dhc\" (UID: \"79bd5ce1-cb17-4035-984a-dfc16c53b637\") " pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:52.101092 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.101053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:52.103931 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.103898    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:52.114510 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.114488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:52.125137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.125114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n9s\" (UniqueName: \"kubernetes.io/projected/45394a99-f247-4de3-aabb-ce4fd06ed5f0-kube-api-access-n4n9s\") pod \"network-check-target-xxl5g\" (UID: \"45394a99-f247-4de3-aabb-ce4fd06ed5f0\") " pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:52.176808 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.176713    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt"]
Sep 05 02:18:52.180612 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:18:52.180587    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5650fe_be9b_4c6f_8428_8193261874ad.slice/crio-03f2d97daec21fd31e9119b41bbbcd8badeaecf692ed73ba76cc71f6574627f9 WatchSource:0}: Error finding container 03f2d97daec21fd31e9119b41bbbcd8badeaecf692ed73ba76cc71f6574627f9: Status 404 returned error can't find the container with id 03f2d97daec21fd31e9119b41bbbcd8badeaecf692ed73ba76cc71f6574627f9
Sep 05 02:18:52.193426 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.193406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dhc"
Sep 05 02:18:52.202296 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.202051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:52.202296 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.202192    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:52.202296 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.202229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:52.202296 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.202258    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.202238846 +0000 UTC m=+67.524011344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:52.202574 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.202310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hwbs\""
Sep 05 02:18:52.202574 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.202337    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:52.202675 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.202582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.202562105 +0000 UTC m=+67.524334616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:52.210103 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.210079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:52.403962 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.403927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:18:52.404347 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.404108    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.404347 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.404176    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls podName:33c51d59-1ee4-456c-acdf-6f4471457da0 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.404159075 +0000 UTC m=+98.725931575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls") pod "dns-default-7p6tl" (UID: "33c51d59-1ee4-456c-acdf-6f4471457da0") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.420020 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.419993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:52.486385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.486356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8dhc"]
Sep 05 02:18:52.488694 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:18:52.488662    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bd5ce1_cb17_4035_984a_dfc16c53b637.slice/crio-909131ab4597e71bdad48f70403df3b09ddd919c7b37335c012a09f046a52c6e WatchSource:0}: Error finding container 909131ab4597e71bdad48f70403df3b09ddd919c7b37335c012a09f046a52c6e: Status 404 returned error can't find the container with id 909131ab4597e71bdad48f70403df3b09ddd919c7b37335c012a09f046a52c6e
Sep 05 02:18:52.504685 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.504661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxl5g"]
Sep 05 02:18:52.508112 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:18:52.508090    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45394a99_f247_4de3_aabb_ce4fd06ed5f0.slice/crio-7792eefb23c78659c59d8eb5126e8ea9a3052c988cba656d87c181aeb1908ebb WatchSource:0}: Error finding container 7792eefb23c78659c59d8eb5126e8ea9a3052c988cba656d87c181aeb1908ebb: Status 404 returned error can't find the container with id 7792eefb23c78659c59d8eb5126e8ea9a3052c988cba656d87c181aeb1908ebb
Sep 05 02:18:52.548693 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.548658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt" event={"ID":"3a5650fe-be9b-4c6f-8428-8193261874ad","Type":"ContainerStarted","Data":"03f2d97daec21fd31e9119b41bbbcd8badeaecf692ed73ba76cc71f6574627f9"}
Sep 05 02:18:52.549841 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.549811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxl5g" event={"ID":"45394a99-f247-4de3-aabb-ce4fd06ed5f0","Type":"ContainerStarted","Data":"7792eefb23c78659c59d8eb5126e8ea9a3052c988cba656d87c181aeb1908ebb"}
Sep 05 02:18:52.550942 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.550922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8dhc" event={"ID":"79bd5ce1-cb17-4035-984a-dfc16c53b637","Type":"ContainerStarted","Data":"909131ab4597e71bdad48f70403df3b09ddd919c7b37335c012a09f046a52c6e"}
Sep 05 02:18:52.807206 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.807125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:52.807206 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:52.807161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:18:52.807397 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.807261    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:52.807397 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.807286    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.807268866 +0000 UTC m=+75.129041366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:52.807397 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:52.807308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.807302179 +0000 UTC m=+75.129074675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : secret "router-metrics-certs-default" not found
Sep 05 02:18:53.210442 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.210401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:53.210657 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:53.210505    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:53.210657 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.210517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:53.210657 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:53.210587    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:55.210565107 +0000 UTC m=+69.532337605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:53.210657 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:53.210648    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:53.210911 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:53.210687    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:55.210676221 +0000 UTC m=+69.532448729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:53.419410 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.419392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lclfz_0b1f3509-edfb-416a-b956-b469a9841b91/dns-node-resolver/0.log"
Sep 05 02:18:53.556829 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.556735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt" event={"ID":"3a5650fe-be9b-4c6f-8428-8193261874ad","Type":"ContainerStarted","Data":"82b12a9b2f1a440917c1166419f4cee0afb91c2e92c848ce84d5d7017564de48"}
Sep 05 02:18:53.556829 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.556817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt" event={"ID":"3a5650fe-be9b-4c6f-8428-8193261874ad","Type":"ContainerStarted","Data":"1091f87835e3d3e6eb5d4183063b15f72aa2bd4fabcf4754a588e16b0b16812d"}
Sep 05 02:18:53.574672 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:53.574600    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hw5qt" podStartSLOduration=1.464288835 podStartE2EDuration="2.574579437s" totalImagesPullingTime="1.110290602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:53.573539836 +0000 UTC m=+67.895312373" watchObservedRunningTime="2026-09-05 02:18:53.574579437 +0000 UTC m=+67.896351957"
Sep 05 02:18:54.621096 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:54.621067    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bh2bf_629c6726-442d-4ee3-99a5-323e477b1e53/node-ca/0.log"
Sep 05 02:18:55.058832 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.058734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:55.159293 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.159218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:55.228236 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.228195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:55.228432 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.228289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:55.228432 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:55.228339    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:55.228432 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:55.228411    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.228394358 +0000 UTC m=+73.550166864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:55.228598 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:55.228432    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:55.228598 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:55.228484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.228468151 +0000 UTC m=+73.550240661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:55.420041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.419995    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8tn8_206465c0-c8d6-4238-b7aa-14409ee968f1/serve-healthcheck-canary/0.log"
Sep 05 02:18:55.956779 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:55.956656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:56.056695 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:56.056658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:56.568417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:56.568375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxl5g" event={"ID":"45394a99-f247-4de3-aabb-ce4fd06ed5f0","Type":"ContainerStarted","Data":"dc1808b7ad7825e90a713f7eb52216938cd58d6f49d8079970f67f675ce6e6fd"}
Sep 05 02:18:56.568591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:56.568538    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:18:56.585432 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:56.585369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xxl5g" podStartSLOduration=67.393548101 podStartE2EDuration="1m10.585353659s" totalImagesPullingTime="3.191805558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:56.584617388 +0000 UTC m=+70.906389922" watchObservedRunningTime="2026-09-05 02:18:56.585353659 +0000 UTC m=+70.907126177"
Sep 05 02:18:56.952370 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:56.952330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.152201 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.152166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.376105 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.376069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.561389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.561297    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.573635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.573594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8dhc" event={"ID":"79bd5ce1-cb17-4035-984a-dfc16c53b637","Type":"ContainerStarted","Data":"d6032a260fa6b42ef9370056e12e930b453bfe4a44da73738c4915bcb6ef866a"}
Sep 05 02:18:57.591151 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.591095    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w8dhc" podStartSLOduration=66.818036246 podStartE2EDuration="1m11.591081555s" totalImagesPullingTime="4.773045309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:57.590656728 +0000 UTC m=+71.912429256" watchObservedRunningTime="2026-09-05 02:18:57.591081555 +0000 UTC m=+71.912854072"
Sep 05 02:18:57.807838 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:57.807798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.589395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:58.589362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.748927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:58.748883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.849459 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:58.849434    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.956491 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:58.956462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:59.025164 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.025129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:59.262244 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.262152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:59.262244 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.262212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:18:59.262437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:59.262280    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.262437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:59.262312    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.262437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:59.262348    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.262332198 +0000 UTC m=+81.584104695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.262437 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:18:59.262364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls podName:bc698016-9978-4b50-9bb0-b80f78de1181 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.262356999 +0000 UTC m=+81.584129494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6f75" (UID: "bc698016-9978-4b50-9bb0-b80f78de1181") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.359952 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.359921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:59.460420 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.460383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:59.561657 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.561562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:59.978282 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:18:59.978248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:00.608716 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:00.608674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:00.876314 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:00.876218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:00.876314 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:00.876264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:00.876496 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:19:00.876411    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle podName:131bce09-5bd6-4376-9aff-440919804a4d nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.876390678 +0000 UTC m=+91.198163177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle") pod "router-default-6c954d494-7nrt6" (UID: "131bce09-5bd6-4376-9aff-440919804a4d") : configmap references non-existent config key: service-ca.crt
Sep 05 02:19:00.878905 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:00.878885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/131bce09-5bd6-4376-9aff-440919804a4d-metrics-certs\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:00.959396 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:00.959359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.161960 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:01.161878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.329895 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:01.329858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.581948 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:01.581864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:02.339180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:02.339152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:02.551210 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:02.551180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:02.637447 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:02.637417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:03.020828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:03.020745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:03.153282 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:03.153240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:03.586383 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:03.586354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.154182 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:04.154149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.673140 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:04.673112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.774421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:04.774389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:05.158115 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:05.158078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:05.568380 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:05.568287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:05.776803 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:05.776748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.322933 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.322794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:19:07.322933 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.322865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:19:07.323441 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.323353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc698016-9978-4b50-9bb0-b80f78de1181-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:19:07.325423 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.325400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc698016-9978-4b50-9bb0-b80f78de1181-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6f75\" (UID: \"bc698016-9978-4b50-9bb0-b80f78de1181\") " pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:19:07.436989 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.436949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6f75"
Sep 05 02:19:07.707135 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:07.707103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6f75"]
Sep 05 02:19:07.710431 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:07.710399    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc698016_9978_4b50_9bb0_b80f78de1181.slice/crio-ee516346b465daa90616e153d1e90bba06e5331ca50c7174e9891f73151fca9f WatchSource:0}: Error finding container ee516346b465daa90616e153d1e90bba06e5331ca50c7174e9891f73151fca9f: Status 404 returned error can't find the container with id ee516346b465daa90616e153d1e90bba06e5331ca50c7174e9891f73151fca9f
Sep 05 02:19:08.381166 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:08.381143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:08.490920 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:08.490890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:08.611994 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:08.611964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6f75" event={"ID":"bc698016-9978-4b50-9bb0-b80f78de1181","Type":"ContainerStarted","Data":"844ec67309d415270a48a9bdbee1213c01513a18fbe48e509cf98336daef0591"}
Sep 05 02:19:08.612151 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:08.612017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6f75" event={"ID":"bc698016-9978-4b50-9bb0-b80f78de1181","Type":"ContainerStarted","Data":"8ea80eac031e4d4357028f0c7367676ed4c28327554ba76fb05a73db04a5b472"}
Sep 05 02:19:08.612151 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:08.612037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6f75" event={"ID":"bc698016-9978-4b50-9bb0-b80f78de1181","Type":"ContainerStarted","Data":"ee516346b465daa90616e153d1e90bba06e5331ca50c7174e9891f73151fca9f"}
Sep 05 02:19:10.623327 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:10.623290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6f75" event={"ID":"bc698016-9978-4b50-9bb0-b80f78de1181","Type":"ContainerStarted","Data":"806bdc46efaa79eb174c9729f38e3d7b052b3adf831dabfd5a4a72cdf7d380e2"}
Sep 05 02:19:10.864591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:10.864549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:10.983652 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:10.983585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.102666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:11.102637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:12.201749 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:12.201718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:12.299647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:12.299616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:12.446685 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:12.446655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:12.695455 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:12.695424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:12.885423 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:12.885368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:13.109885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:13.109855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:13.210590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:13.210562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.116970 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.116916    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c6f75" podStartSLOduration=21.11569645 podStartE2EDuration="23.116899416s" totalImagesPullingTime="2.001202966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:10.66748426 +0000 UTC m=+84.989256777" watchObservedRunningTime="2026-09-05 02:19:14.116899416 +0000 UTC m=+88.438671933"
Sep 05 02:19:14.117715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.117697    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"]
Sep 05 02:19:14.121002 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.120988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.124255 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.124233    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 05 02:19:14.125282 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 05 02:19:14.125385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125306    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 05 02:19:14.125385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:14.125385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125311    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 05 02:19:14.125513 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125438    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:14.125513 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.125490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:14.159877 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.159848    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"]
Sep 05 02:19:14.172019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.171991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.172128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.172022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.172128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.172044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.172128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.172069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxrp\" (UniqueName: \"kubernetes.io/projected/a2a848e6-6696-40dd-8cc0-e484d765432d-kube-api-access-4fxrp\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.172128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.172106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.172287 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.172133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a848e6-6696-40dd-8cc0-e484d765432d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.210971 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.210938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.272562 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a848e6-6696-40dd-8cc0-e484d765432d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.272743 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.272743 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.272743 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.272743 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxrp\" (UniqueName: \"kubernetes.io/projected/a2a848e6-6696-40dd-8cc0-e484d765432d-kube-api-access-4fxrp\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.272955 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.272844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.273425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.273398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a848e6-6696-40dd-8cc0-e484d765432d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.275334 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.275304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-ca\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.275334 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.275316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.275481 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.275463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.275584 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.275566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2a848e6-6696-40dd-8cc0-e484d765432d-hub\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.282685 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.282663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxrp\" (UniqueName: \"kubernetes.io/projected/a2a848e6-6696-40dd-8cc0-e484d765432d-kube-api-access-4fxrp\") pod \"cluster-proxy-proxy-agent-788d46755c-sn4mx\" (UID: \"a2a848e6-6696-40dd-8cc0-e484d765432d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.331518 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.331469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.433026 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.432936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.449873 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.449839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"
Sep 05 02:19:14.531928 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.531901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.630889 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.630856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:14.728318 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:14.728196    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx"]
Sep 05 02:19:14.731842 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:14.731815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2a848e6_6696_40dd_8cc0_e484d765432d.slice/crio-b999e9afa19c6705eae41e6e1b4c6fad7151ee456dea63ebace29a3181b46e41 WatchSource:0}: Error finding container b999e9afa19c6705eae41e6e1b4c6fad7151ee456dea63ebace29a3181b46e41: Status 404 returned error can't find the container with id b999e9afa19c6705eae41e6e1b4c6fad7151ee456dea63ebace29a3181b46e41
Sep 05 02:19:15.641034 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:15.640988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx" event={"ID":"a2a848e6-6696-40dd-8cc0-e484d765432d","Type":"ContainerStarted","Data":"b999e9afa19c6705eae41e6e1b4c6fad7151ee456dea63ebace29a3181b46e41"}
Sep 05 02:19:16.892867 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:16.892827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:16.893533 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:16.893487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131bce09-5bd6-4376-9aff-440919804a4d-service-ca-bundle\") pod \"router-default-6c954d494-7nrt6\" (UID: \"131bce09-5bd6-4376-9aff-440919804a4d\") " pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:17.072368 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:17.072330    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j86s6\""
Sep 05 02:19:17.079465 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:17.079437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:18.376860 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.376832    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c954d494-7nrt6"]
Sep 05 02:19:18.379902 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:18.379872    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131bce09_5bd6_4376_9aff_440919804a4d.slice/crio-c1a34418c9b73fb321c98f91ebadee5bda5a4da76a8672bc9330f34d86356e19 WatchSource:0}: Error finding container c1a34418c9b73fb321c98f91ebadee5bda5a4da76a8672bc9330f34d86356e19: Status 404 returned error can't find the container with id c1a34418c9b73fb321c98f91ebadee5bda5a4da76a8672bc9330f34d86356e19
Sep 05 02:19:18.658357 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.658260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx" event={"ID":"a2a848e6-6696-40dd-8cc0-e484d765432d","Type":"ContainerStarted","Data":"5155d4e6b0128609dc8db056d2d3344ad776fc7b7d0a2ce5dc3380a9aa7d2422"}
Sep 05 02:19:18.658357 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.658320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx" event={"ID":"a2a848e6-6696-40dd-8cc0-e484d765432d","Type":"ContainerStarted","Data":"e544760e5dc244c2aa42d0d046683dc3355b33626834d3dacbb8fdcee8f46a27"}
Sep 05 02:19:18.658357 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.658341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx" event={"ID":"a2a848e6-6696-40dd-8cc0-e484d765432d","Type":"ContainerStarted","Data":"4318fd648c32343b64f1d0b048bbbdc80a3cf026808cd10ebe434bb13af398cc"}
Sep 05 02:19:18.659934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.659908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c954d494-7nrt6" event={"ID":"131bce09-5bd6-4376-9aff-440919804a4d","Type":"ContainerStarted","Data":"9b37526f68f49de8571db3a26501b11a941f64fbb2f0207343babea99037fb4f"}
Sep 05 02:19:18.660053 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.659972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c954d494-7nrt6" event={"ID":"131bce09-5bd6-4376-9aff-440919804a4d","Type":"ContainerStarted","Data":"c1a34418c9b73fb321c98f91ebadee5bda5a4da76a8672bc9330f34d86356e19"}
Sep 05 02:19:18.682930 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.682873    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-788d46755c-sn4mx" podStartSLOduration=1.24226248 podStartE2EDuration="4.682860176s" totalImagesPullingTime="3.440597696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:18.681497672 +0000 UTC m=+93.003270188" watchObservedRunningTime="2026-09-05 02:19:18.682860176 +0000 UTC m=+93.004632694"
Sep 05 02:19:18.702239 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.702169    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c954d494-7nrt6" podStartSLOduration=34.702149247 podStartE2EDuration="34.702149247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:18.701942475 +0000 UTC m=+93.023715007" watchObservedRunningTime="2026-09-05 02:19:18.702149247 +0000 UTC m=+93.023921760"
Sep 05 02:19:18.707646 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.707609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:18.806338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:18.806302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:19.080398 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:19.080313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:19.083031 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:19.083008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:19.663346 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:19.663309    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:19.664614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:19.664592    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c954d494-7nrt6"
Sep 05 02:19:21.119565 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:21.119522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:21.219142 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:21.219111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:21.300023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:21.299991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:22.067904 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:22.067853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:23.713681 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:23.713636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:23.774135 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:23.774106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:24.452343 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.452307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:19:24.454658 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.454631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33c51d59-1ee4-456c-acdf-6f4471457da0-metrics-tls\") pod \"dns-default-7p6tl\" (UID: \"33c51d59-1ee4-456c-acdf-6f4471457da0\") " pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:19:24.488107 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.488079    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l69pm\""
Sep 05 02:19:24.496046 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.496020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:19:24.680118 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.680082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:24.737715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.737651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:24.771550 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:24.771512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7p6tl"]
Sep 05 02:19:24.774730 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:24.774700    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c51d59_1ee4_456c_acdf_6f4471457da0.slice/crio-bcccd1f636c1b2ae27de1c328234c286cf7a59c1f71a7dc78d8aecdbf1d12ed2 WatchSource:0}: Error finding container bcccd1f636c1b2ae27de1c328234c286cf7a59c1f71a7dc78d8aecdbf1d12ed2: Status 404 returned error can't find the container with id bcccd1f636c1b2ae27de1c328234c286cf7a59c1f71a7dc78d8aecdbf1d12ed2
Sep 05 02:19:25.682548 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:25.682517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7p6tl" event={"ID":"33c51d59-1ee4-456c-acdf-6f4471457da0","Type":"ContainerStarted","Data":"bcccd1f636c1b2ae27de1c328234c286cf7a59c1f71a7dc78d8aecdbf1d12ed2"}
Sep 05 02:19:26.138175 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.138138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.204922 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.204873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.687922 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.687830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7p6tl" event={"ID":"33c51d59-1ee4-456c-acdf-6f4471457da0","Type":"ContainerStarted","Data":"b6861b4fbc1819a3485b8a126dcb2911a8c8af397e24f674904f872c8dc9d50e"}
Sep 05 02:19:26.687922 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.687893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7p6tl" event={"ID":"33c51d59-1ee4-456c-acdf-6f4471457da0","Type":"ContainerStarted","Data":"f97ed977c431f402e49a9fe81a39f85d69a55f027a1698bfabfd7d8a809fded6"}
Sep 05 02:19:26.688123 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.687985    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:19:26.706936 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:26.706887    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7p6tl" podStartSLOduration=65.252661002 podStartE2EDuration="1m6.706871999s" totalImagesPullingTime="1.454210997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:26.705200821 +0000 UTC m=+101.026973339" watchObservedRunningTime="2026-09-05 02:19:26.706871999 +0000 UTC m=+101.028644517"
Sep 05 02:19:27.575901 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:27.575871    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xxl5g"
Sep 05 02:19:28.302325 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:28.302295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:28.364804 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:28.364753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:29.918715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:29.918682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:30.027241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:30.027208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.281362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:33.281330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.382931 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:33.382899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.928781 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:33.928737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.994257 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:33.994228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:34.388656 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:34.388628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.299647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:35.299615    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.358825 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:35.358797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.918827 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:35.918786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:36.019619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:36.019584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:36.119735 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:36.119699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:36.694210 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:36.694180    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7p6tl"
Sep 05 02:19:37.013977 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:37.013874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:37.081387 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:37.081357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:37.404980 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:37.404948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:37.505740 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:37.505710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:38.403885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:38.403856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:38.517035 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:38.517007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.479013 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.478978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.699750 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.699715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.814934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.814845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.833513 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.833476    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pm695"]
Sep 05 02:19:39.835627 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.835603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.845369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.845347    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:39.846156 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:39.846246 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846170    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:39.846307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846243    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:39.846307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4v597\""
Sep 05 02:19:39.846307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846290    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:39.846448 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.846358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:39.914555 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.914517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.967649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-wtmp\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.967863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-textfile\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.967863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.967863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdv6g\" (UniqueName: \"kubernetes.io/projected/203d1a45-4272-45a2-b07e-9d246b1aa368-kube-api-access-sdv6g\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.967863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.968038 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-sys\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.968038 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-metrics-client-ca\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.968038 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-root\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:39.968038 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:39.967939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068469 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-root\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068469 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-root\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-wtmp\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-textfile\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdv6g\" (UniqueName: \"kubernetes.io/projected/203d1a45-4272-45a2-b07e-9d246b1aa368-kube-api-access-sdv6g\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.068700 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-wtmp\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-sys\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-metrics-client-ca\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:19:40.068774    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:19:40.068851    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls podName:203d1a45-4272-45a2-b07e-9d246b1aa368 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:40.56883307 +0000 UTC m=+114.890605579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls") pod "node-exporter-pm695" (UID: "203d1a45-4272-45a2-b07e-9d246b1aa368") : secret "node-exporter-tls" not found
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-textfile\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069021 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.068995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069264 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.069043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/203d1a45-4272-45a2-b07e-9d246b1aa368-sys\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.069299 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.069278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203d1a45-4272-45a2-b07e-9d246b1aa368-metrics-client-ca\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.071298 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.071267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.092395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.092360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdv6g\" (UniqueName: \"kubernetes.io/projected/203d1a45-4272-45a2-b07e-9d246b1aa368-kube-api-access-sdv6g\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.573316 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.573272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.575777 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.575741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/203d1a45-4272-45a2-b07e-9d246b1aa368-node-exporter-tls\") pod \"node-exporter-pm695\" (UID: \"203d1a45-4272-45a2-b07e-9d246b1aa368\") " pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.745262 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:40.745229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pm695"
Sep 05 02:19:40.754932 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:40.754904    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203d1a45_4272_45a2_b07e_9d246b1aa368.slice/crio-1c52f993bfadc59c21bae5b1c84ab55ed15bc65494f982c07fbfde30e5a12ee7 WatchSource:0}: Error finding container 1c52f993bfadc59c21bae5b1c84ab55ed15bc65494f982c07fbfde30e5a12ee7: Status 404 returned error can't find the container with id 1c52f993bfadc59c21bae5b1c84ab55ed15bc65494f982c07fbfde30e5a12ee7
Sep 05 02:19:41.315900 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.315858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:41.410513 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.410485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:41.738550 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.738517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:41.739645 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.739619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm695" event={"ID":"203d1a45-4272-45a2-b07e-9d246b1aa368","Type":"ContainerStarted","Data":"d99063ff4d51cf2353f4e7dbe267b6bf4cf48907b14cb92e32693a3c53fe9b50"}
Sep 05 02:19:41.739805 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.739681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm695" event={"ID":"203d1a45-4272-45a2-b07e-9d246b1aa368","Type":"ContainerStarted","Data":"1c52f993bfadc59c21bae5b1c84ab55ed15bc65494f982c07fbfde30e5a12ee7"}
Sep 05 02:19:41.864422 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:41.864391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:42.745215 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:42.745180    2582 generic.go:356] "Generic (PLEG): container finished" podID="203d1a45-4272-45a2-b07e-9d246b1aa368" containerID="d99063ff4d51cf2353f4e7dbe267b6bf4cf48907b14cb92e32693a3c53fe9b50" exitCode=0
Sep 05 02:19:42.745584 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:42.745250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm695" event={"ID":"203d1a45-4272-45a2-b07e-9d246b1aa368","Type":"ContainerDied","Data":"d99063ff4d51cf2353f4e7dbe267b6bf4cf48907b14cb92e32693a3c53fe9b50"}
Sep 05 02:19:42.775840 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:42.775812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:42.881487 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:42.881459    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:43.333449 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:43.333396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:43.529715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:43.529687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:43.751077 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:43.751041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm695" event={"ID":"203d1a45-4272-45a2-b07e-9d246b1aa368","Type":"ContainerStarted","Data":"1eaaea90138f32c97da36851e1e977c82545e227a9aec75ba793856db30a9b9f"}
Sep 05 02:19:43.751452 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:43.751133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm695" event={"ID":"203d1a45-4272-45a2-b07e-9d246b1aa368","Type":"ContainerStarted","Data":"8db78d71965eef08c7fca7ce240bf84535e14bde5f9d192b87c895ec83235f91"}
Sep 05 02:19:43.770666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:43.770617    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pm695" podStartSLOduration=3.824295889 podStartE2EDuration="4.770604402s" totalImagesPullingTime="875.27819ms" totalInitContainerRuntime="71.030323ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:43.76971914 +0000 UTC m=+118.091491676" watchObservedRunningTime="2026-09-05 02:19:43.770604402 +0000 UTC m=+118.092376920"
Sep 05 02:19:44.515526 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:44.515491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:44.596897 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:44.596854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:45.196875 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:45.196840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:45.843146 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:45.843114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:45.969751 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:45.969716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:46.067373 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:46.067347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:46.162151 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:46.162062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:49.194727 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:49.194689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:49.294568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:49.294532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:49.373959 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:49.373930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:53.425643 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:53.425595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:53.500830 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:53.500794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:55.993144 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:55.993089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:19:55.995595 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:55.995574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c79c34-5580-45bb-85b2-b82afd3dd6e2-metrics-certs\") pod \"network-metrics-daemon-2nhpg\" (UID: \"37c79c34-5580-45bb-85b2-b82afd3dd6e2\") " pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:19:56.108448 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.108413    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66tlm\""
Sep 05 02:19:56.116168 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.116137    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2nhpg"
Sep 05 02:19:56.287849 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.287745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:56.390871 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.390838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2nhpg"]
Sep 05 02:19:56.394303 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:19:56.394269    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c79c34_5580_45bb_85b2_b82afd3dd6e2.slice/crio-c5fe454f4a2627b4ef839e1add5a7ae2180f1726bdf4f6a7f02ea572cec73acc WatchSource:0}: Error finding container c5fe454f4a2627b4ef839e1add5a7ae2180f1726bdf4f6a7f02ea572cec73acc: Status 404 returned error can't find the container with id c5fe454f4a2627b4ef839e1add5a7ae2180f1726bdf4f6a7f02ea572cec73acc
Sep 05 02:19:56.397012 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.396981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:56.490156 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.490124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:56.795250 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:56.795208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2nhpg" event={"ID":"37c79c34-5580-45bb-85b2-b82afd3dd6e2","Type":"ContainerStarted","Data":"c5fe454f4a2627b4ef839e1add5a7ae2180f1726bdf4f6a7f02ea572cec73acc"}
Sep 05 02:19:57.800373 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:57.800341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2nhpg" event={"ID":"37c79c34-5580-45bb-85b2-b82afd3dd6e2","Type":"ContainerStarted","Data":"a2d184d3ba0e8d842447834a23219ba117f1ad53bc385c4b39e714d7eb238100"}
Sep 05 02:19:57.800757 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:57.800391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2nhpg" event={"ID":"37c79c34-5580-45bb-85b2-b82afd3dd6e2","Type":"ContainerStarted","Data":"432dd035874da71f8c01160930592d909427f689ec9069890917f7dba0343140"}
Sep 05 02:19:57.817629 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:57.817575    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2nhpg" podStartSLOduration=130.968665298 podStartE2EDuration="2m11.817560932s" totalImagesPullingTime="848.895634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:57.816678915 +0000 UTC m=+132.138451441" watchObservedRunningTime="2026-09-05 02:19:57.817560932 +0000 UTC m=+132.139333450"
Sep 05 02:19:57.914137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:57.914105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:57.987071 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:19:57.987041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:05.027225 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:05.027191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:05.115119 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:05.115087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.416335 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:08.416306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.537688 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:08.537611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.838245 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:08.838158    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f7ef39f-8036-4d72-9221-fb17307f199d" containerID="99e952954e90dd32e8fa60c01962e2a9e5c4f8d15be3ae5f390f48f5d9f5cd6d" exitCode=0
Sep 05 02:20:08.838245 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:08.838214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng" event={"ID":"4f7ef39f-8036-4d72-9221-fb17307f199d","Type":"ContainerDied","Data":"99e952954e90dd32e8fa60c01962e2a9e5c4f8d15be3ae5f390f48f5d9f5cd6d"}
Sep 05 02:20:08.838502 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:08.838490    2582 scope.go:117] "RemoveContainer" containerID="99e952954e90dd32e8fa60c01962e2a9e5c4f8d15be3ae5f390f48f5d9f5cd6d"
Sep 05 02:20:09.843450 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:09.843416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-487ng" event={"ID":"4f7ef39f-8036-4d72-9221-fb17307f199d","Type":"ContainerStarted","Data":"586437a79fef4a2630c99da3289a909f565f33c5241aa0d1fcf3cf6438b9b6ed"}
Sep 05 02:20:59.137654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:59.137564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:59.536124 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:20:59.536040    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:08.593357 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:08.593327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:08.663481 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:08.663446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:08.736044 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:08.736013    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:08.959525 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:08.959440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:09.341036 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:09.340952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:14.365852 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:14.365820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:14.466928 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:14.466898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:26.242134 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:26.242106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:26.299671 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:26.299636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:28.362029 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:28.361991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:28.696007 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:28.695916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:28.842241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:28.842197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:28.943741 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:28.943706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:38.395640 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:38.395610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:38.496477 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:38.496449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:47.371813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:47.371776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:47.436884 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:21:47.436850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:46.119452 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:22:46.119424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:22:46.123284 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:22:46.123246    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:22:46.123746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:22:46.123729    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:23:12.625464 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:12.625428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:12.682249 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:12.682214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:16.127268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:16.127236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:17.678406 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:17.678379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:28.570557 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:28.570524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:38.502072 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:38.501986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:38.605472 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:38.605441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:44.422948 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:44.422909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:44.707392 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:44.707304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:44.768007 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:44.767975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:44.887840 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:44.887802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:45.408245 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:45.408216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:45.511009 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:45.510975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:45.607367 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:45.607338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:45.869972 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:45.869939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:45.970920 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:45.970889    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:46.198541 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:46.198453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:46.298804 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:46.298741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:46.424013 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:46.423980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:46.606224 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:23:46.606195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:26.926506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:26.926468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:32.321889 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:32.321848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:32.398404 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:32.398371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:32.471036 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:32.471002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.044014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:33.043982    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.112453 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:33.112423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:37.253421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:37.253380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:37.369439 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:37.369409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:43.568611 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:43.568569    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:43.672842 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:24:43.672808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.577888 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.577837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.651290 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.651254    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:06.654253 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.654238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.656487 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.656463    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:25:06.657304 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.657283    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 05 02:25:06.657407 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.657308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:25:06.657407 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.657313    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-r5dqp\""
Sep 05 02:25:06.659619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.659597    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"]
Sep 05 02:25:06.662662 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.662641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.664643 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.664626    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 05 02:25:06.664801 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.664784    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-s9p95\""
Sep 05 02:25:06.669231 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.669213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:06.676105 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.676082    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"]
Sep 05 02:25:06.677180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.677156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.766098 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.766065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.836136 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.836053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.838043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.838025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-kube-api-access-whrg7\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.838126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.838060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwrj\" (UniqueName: \"kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.838126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.838087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.838197 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.838178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.939487 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.939451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.939647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.939497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.939647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.939622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-kube-api-access-whrg7\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.939718 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.939667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwrj\" (UniqueName: \"kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.942134 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.942108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.942243 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.942108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.945572 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.945543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:06.948198 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.948175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/c3d9108f-ba55-4ff7-a565-eed6d0289c0a-kube-api-access-whrg7\") pod \"llmisvc-controller-manager-645f9c8fd6-9v27f\" (UID: \"c3d9108f-ba55-4ff7-a565-eed6d0289c0a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:06.948304 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.948223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwrj\" (UniqueName: \"kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj\") pod \"kserve-controller-manager-577b5fb6f8-rx5db\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.965002 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.964984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:06.971913 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:06.971891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:07.244839 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.244735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:07.247728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.247703    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"]
Sep 05 02:25:07.248064 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:25:07.248044    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b220dcc_9f03_4d1d_a1d7_a91188ae1f18.slice/crio-279f783a29db12bdf454e60f4b8cf79eedb9062f8931ad8bc3d10f6fcfd68f1c WatchSource:0}: Error finding container 279f783a29db12bdf454e60f4b8cf79eedb9062f8931ad8bc3d10f6fcfd68f1c: Status 404 returned error can't find the container with id 279f783a29db12bdf454e60f4b8cf79eedb9062f8931ad8bc3d10f6fcfd68f1c
Sep 05 02:25:07.249536 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.249520    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:25:07.251241 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:25:07.251219    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc3d9108f_ba55_4ff7_a565_eed6d0289c0a.slice/crio-0c1f88c08a736dc65fd724f5bcc30da5b2f9f62b5a6902363a262ab948941866 WatchSource:0}: Error finding container 0c1f88c08a736dc65fd724f5bcc30da5b2f9f62b5a6902363a262ab948941866: Status 404 returned error can't find the container with id 0c1f88c08a736dc65fd724f5bcc30da5b2f9f62b5a6902363a262ab948941866
Sep 05 02:25:07.541681 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.541589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:07.639551 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.639518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:07.828653 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.828511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f" event={"ID":"c3d9108f-ba55-4ff7-a565-eed6d0289c0a","Type":"ContainerStarted","Data":"0c1f88c08a736dc65fd724f5bcc30da5b2f9f62b5a6902363a262ab948941866"}
Sep 05 02:25:07.830175 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:07.830140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" event={"ID":"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18","Type":"ContainerStarted","Data":"279f783a29db12bdf454e60f4b8cf79eedb9062f8931ad8bc3d10f6fcfd68f1c"}
Sep 05 02:25:10.842097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.841986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f" event={"ID":"c3d9108f-ba55-4ff7-a565-eed6d0289c0a","Type":"ContainerStarted","Data":"c514fa36b7aa60df6dbc1a5b9921271acd320befe58fda76a76cbb56d433842e"}
Sep 05 02:25:10.842635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.842131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:10.843687 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.843661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" event={"ID":"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18","Type":"ContainerStarted","Data":"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"}
Sep 05 02:25:10.843821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.843807    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:10.858625 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.858585    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f" podStartSLOduration=1.6508534639999999 podStartE2EDuration="4.858572361s" totalImagesPullingTime="3.207718897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:10.857685543 +0000 UTC m=+445.179458082" watchObservedRunningTime="2026-09-05 02:25:10.858572361 +0000 UTC m=+445.180344880"
Sep 05 02:25:10.876302 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:10.876215    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" podStartSLOduration=1.66362246 podStartE2EDuration="4.876195066s" totalImagesPullingTime="3.212572606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:10.875798499 +0000 UTC m=+445.197571027" watchObservedRunningTime="2026-09-05 02:25:10.876195066 +0000 UTC m=+445.197967576"
Sep 05 02:25:41.848897 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:41.848864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9v27f"
Sep 05 02:25:41.851862 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:41.851844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:55.472607 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.472578    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:55.473015 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.472789    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" podUID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" containerName="manager" containerID="cri-o://57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca" gracePeriod=10
Sep 05 02:25:55.528403 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.528367    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-cnxhl"]
Sep 05 02:25:55.535705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.535681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.550312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.550282    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-cnxhl"]
Sep 05 02:25:55.697935 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.697908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbclv\" (UniqueName: \"kubernetes.io/projected/6007671b-bfff-487e-bdf5-6e8123379edf-kube-api-access-mbclv\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.698028 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.697948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6007671b-bfff-487e-bdf5-6e8123379edf-cert\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.707969 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.707948    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:55.798670 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.798590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqwrj\" (UniqueName: \"kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj\") pod \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") "
Sep 05 02:25:55.798670 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.798625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert\") pod \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\" (UID: \"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18\") "
Sep 05 02:25:55.798877 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.798718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbclv\" (UniqueName: \"kubernetes.io/projected/6007671b-bfff-487e-bdf5-6e8123379edf-kube-api-access-mbclv\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.798877 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.798772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6007671b-bfff-487e-bdf5-6e8123379edf-cert\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.801019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.800990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert" (OuterVolumeSpecName: "cert") pod "6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" (UID: "6b220dcc-9f03-4d1d-a1d7-a91188ae1f18"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:25:55.801137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.801033    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj" (OuterVolumeSpecName: "kube-api-access-hqwrj") pod "6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" (UID: "6b220dcc-9f03-4d1d-a1d7-a91188ae1f18"). InnerVolumeSpecName "kube-api-access-hqwrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:25:55.801137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.801053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6007671b-bfff-487e-bdf5-6e8123379edf-cert\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.811469 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.811450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbclv\" (UniqueName: \"kubernetes.io/projected/6007671b-bfff-487e-bdf5-6e8123379edf-kube-api-access-mbclv\") pod \"kserve-controller-manager-577b5fb6f8-cnxhl\" (UID: \"6007671b-bfff-487e-bdf5-6e8123379edf\") " pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.844551 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.844524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:55.899438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.899406    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwrj\" (UniqueName: \"kubernetes.io/projected/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-kube-api-access-hqwrj\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:25:55.899438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.899437    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:25:55.996978 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.996942    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" containerID="57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca" exitCode=0
Sep 05 02:25:55.996978 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.996984    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db"
Sep 05 02:25:55.997172 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.997029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" event={"ID":"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18","Type":"ContainerDied","Data":"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"}
Sep 05 02:25:55.997172 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.997077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rx5db" event={"ID":"6b220dcc-9f03-4d1d-a1d7-a91188ae1f18","Type":"ContainerDied","Data":"279f783a29db12bdf454e60f4b8cf79eedb9062f8931ad8bc3d10f6fcfd68f1c"}
Sep 05 02:25:55.997172 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:55.997112    2582 scope.go:117] "RemoveContainer" containerID="57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"
Sep 05 02:25:56.006628 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.006613    2582 scope.go:117] "RemoveContainer" containerID="57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"
Sep 05 02:25:56.006949 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:25:56.006929    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca\": container with ID starting with 57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca not found: ID does not exist" containerID="57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"
Sep 05 02:25:56.007003 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.006957    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca"} err="failed to get container status \"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca\": rpc error: code = NotFound desc = could not find container \"57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca\": container with ID starting with 57280bf90862fd7ba67127364afeb020ccae586c112e2f30b35ae4d2dd9cf1ca not found: ID does not exist"
Sep 05 02:25:56.030889 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.030864    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:56.036178 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.036156    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rx5db"]
Sep 05 02:25:56.122739 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.122709    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-cnxhl"]
Sep 05 02:25:56.126233 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:25:56.126212    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6007671b_bfff_487e_bdf5_6e8123379edf.slice/crio-4d03614cdbcab5b9e77c6e49df4531b9773718ca914703cc467465ae3558467c WatchSource:0}: Error finding container 4d03614cdbcab5b9e77c6e49df4531b9773718ca914703cc467465ae3558467c: Status 404 returned error can't find the container with id 4d03614cdbcab5b9e77c6e49df4531b9773718ca914703cc467465ae3558467c
Sep 05 02:25:56.182458 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:56.182429    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" path="/var/lib/kubelet/pods/6b220dcc-9f03-4d1d-a1d7-a91188ae1f18/volumes"
Sep 05 02:25:57.004151 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:57.004114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl" event={"ID":"6007671b-bfff-487e-bdf5-6e8123379edf","Type":"ContainerStarted","Data":"234d7c0af3919f48ea2bae98993a4a4365f583bce8668fe2933e106575b05ba6"}
Sep 05 02:25:57.004554 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:57.004172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl" event={"ID":"6007671b-bfff-487e-bdf5-6e8123379edf","Type":"ContainerStarted","Data":"4d03614cdbcab5b9e77c6e49df4531b9773718ca914703cc467465ae3558467c"}
Sep 05 02:25:57.004554 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:57.004250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:25:57.028637 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:25:57.028593    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl" podStartSLOduration=1.580931688 podStartE2EDuration="2.028578332s" totalImagesPullingTime="447.646644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:57.02825592 +0000 UTC m=+491.350028449" watchObservedRunningTime="2026-09-05 02:25:57.028578332 +0000 UTC m=+491.350350850"
Sep 05 02:26:28.013519 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:28.013490    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-cnxhl"
Sep 05 02:26:29.302638 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:29.302591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:29.385287 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:29.385260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:29.563153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:29.563072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:29.662483 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:29.662450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:29.762822 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:29.762751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:30.473152 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:30.473119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:30.533581 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:26:30.533555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:46.143543 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:27:46.143513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:27:46.145039 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:27:46.145014    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:28:04.243509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:04.243475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:04.321682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:04.321647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:16.741705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:16.741677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:16.966030 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:16.965995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:17.163992 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.163956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:17.267838 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.267807    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:28:17.268069 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.268057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" containerName="manager"
Sep 05 02:28:17.268111 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.268071    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" containerName="manager"
Sep 05 02:28:17.268148 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.268113    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b220dcc-9f03-4d1d-a1d7-a91188ae1f18" containerName="manager"
Sep 05 02:28:17.274625 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.274600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.283169 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.283146    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:28:17.283516 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.283490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 05 02:28:17.283574 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.283544    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:28:17.283647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.283629    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:28:17.283857 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.283837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:28:17.284544 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.284530    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 05 02:28:17.284584 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.284562    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:28:17.308886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.308860    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:28:17.309839 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.309818    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:28:17.313027 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.313007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.316921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.316902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 05 02:28:17.333992 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.333962    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:28:17.394454 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wr8g\" (UniqueName: \"kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.394454 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.394649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.394649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.394649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dft6q\" (UniqueName: \"kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.394649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.394649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.394828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.394828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.394828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.394737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.398982 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.398960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:17.496023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.495946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.496023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.495978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.496023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.495996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.496023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.496291 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wr8g\" (UniqueName: \"kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.496351 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.496407 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.496472 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.496663 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dft6q\" (UniqueName: \"kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.496807 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.496950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.497033 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.497033 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.496860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.497108 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.497076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.497142 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.497103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.497266 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.497244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.498705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.498688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.498820 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.498802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.542583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.542555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dft6q\" (UniqueName: \"kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q\") pod \"isvc-canary-rollback-predictor-76998db6fd-7s427\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.554978 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.554953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wr8g\" (UniqueName: \"kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.584615 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.584584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:17.623522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.623492    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:17.623522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.623542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:17.871443 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.871397    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:28:17.873460 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:28:17.873441    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fe66a8_8b52_446f_9ba1_8dc6e198a353.slice/crio-dbcee21e70b3b777224fb9b5ba43e7bbf4cd8ae5f1095aed04c44143b8635449 WatchSource:0}: Error finding container dbcee21e70b3b777224fb9b5ba43e7bbf4cd8ae5f1095aed04c44143b8635449: Status 404 returned error can't find the container with id dbcee21e70b3b777224fb9b5ba43e7bbf4cd8ae5f1095aed04c44143b8635449
Sep 05 02:28:17.908274 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:17.908245    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:28:17.911354 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:28:17.911330    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb53e845_c89d_47f2_852b_7313c92925e3.slice/crio-a8f854e22caa0709901e5cbf5d839080050201f76f8e3e478f9e3275546f02ce WatchSource:0}: Error finding container a8f854e22caa0709901e5cbf5d839080050201f76f8e3e478f9e3275546f02ce: Status 404 returned error can't find the container with id a8f854e22caa0709901e5cbf5d839080050201f76f8e3e478f9e3275546f02ce
Sep 05 02:28:18.503127 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:18.503069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerStarted","Data":"dbcee21e70b3b777224fb9b5ba43e7bbf4cd8ae5f1095aed04c44143b8635449"}
Sep 05 02:28:18.504707 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:18.504628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerStarted","Data":"a8f854e22caa0709901e5cbf5d839080050201f76f8e3e478f9e3275546f02ce"}
Sep 05 02:28:21.520927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:21.520884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerStarted","Data":"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"}
Sep 05 02:28:21.522802 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:21.522757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerStarted","Data":"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"}
Sep 05 02:28:25.538164 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:25.538125    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb53e845-c89d-47f2-852b-7313c92925e3" containerID="c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac" exitCode=0
Sep 05 02:28:25.538631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:25.538205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerDied","Data":"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"}
Sep 05 02:28:25.539653 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:25.539632    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerID="e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900" exitCode=0
Sep 05 02:28:25.539735 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:25.539666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerDied","Data":"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"}
Sep 05 02:28:41.627778 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:41.627728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerStarted","Data":"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"}
Sep 05 02:28:41.629526 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:41.629501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerStarted","Data":"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"}
Sep 05 02:28:44.646112 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.646073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerStarted","Data":"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"}
Sep 05 02:28:44.646734 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.646145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:44.646734 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.646179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:44.647574 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.647539    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:28:44.648317 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.648296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerStarted","Data":"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"}
Sep 05 02:28:44.648457 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.648442    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:44.667706 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.667661    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podStartSLOduration=2.680533295 podStartE2EDuration="27.667648758s" totalImagesPullingTime="21.211687377s" totalInitContainerRuntime="3.775428086s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:44.66556298 +0000 UTC m=+658.987335498" watchObservedRunningTime="2026-09-05 02:28:44.667648758 +0000 UTC m=+658.989421275"
Sep 05 02:28:44.684062 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:44.684020    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podStartSLOduration=2.777395095 podStartE2EDuration="27.684006269s" totalImagesPullingTime="21.299096892s" totalInitContainerRuntime="3.607514282s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:44.682956575 +0000 UTC m=+659.004729106" watchObservedRunningTime="2026-09-05 02:28:44.684006269 +0000 UTC m=+659.005778787"
Sep 05 02:28:45.652456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:45.652423    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:45.652933 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:45.652468    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:28:45.653610 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:45.653588    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:28:46.656584 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:46.656546    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:28:50.657268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:50.657239    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:28:50.657850 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:50.657821    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:28:51.660673 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:51.660647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:28:51.661096 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:28:51.661071    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:00.658311 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:00.658270    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:01.661744 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:01.661710    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:10.657778 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:10.657730    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:11.661111 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:11.661075    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:20.658017 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:20.657972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:21.661125 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:21.661087    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:30.658163 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:30.658123    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:31.661478 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:31.661422    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:40.658421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:40.658342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:41.661308 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:41.661270    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:29:50.658649 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:50.658614    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 05 02:29:51.661345 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:29:51.661308    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:30:00.658503 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:00.658475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:30:01.662593 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.662567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:30:01.800519 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.800483    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:30:01.800709 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.800688    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container" containerID="cri-o://3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606" gracePeriod=30
Sep 05 02:30:01.800829 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.800786    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kube-rbac-proxy" containerID="cri-o://b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466" gracePeriod=30
Sep 05 02:30:01.936743 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.936710    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb53e845-c89d-47f2-852b-7313c92925e3" containerID="b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466" exitCode=2
Sep 05 02:30:01.936905 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:01.936785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerDied","Data":"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"}
Sep 05 02:30:05.653082 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:05.653038    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Sep 05 02:30:06.644954 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.644928    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:30:06.662348 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.662319    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:30:06.662725 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.662561    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" containerID="cri-o://a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89" gracePeriod=30
Sep 05 02:30:06.662725 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.662638    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kube-rbac-proxy" containerID="cri-o://5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75" gracePeriod=30
Sep 05 02:30:06.732237 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.732193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:06.750975 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.750898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config\") pod \"fb53e845-c89d-47f2-852b-7313c92925e3\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") "
Sep 05 02:30:06.750975 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.750930    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dft6q\" (UniqueName: \"kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q\") pod \"fb53e845-c89d-47f2-852b-7313c92925e3\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") "
Sep 05 02:30:06.750975 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.750951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert\") pod \"fb53e845-c89d-47f2-852b-7313c92925e3\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") "
Sep 05 02:30:06.750975 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.750974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location\") pod \"fb53e845-c89d-47f2-852b-7313c92925e3\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") "
Sep 05 02:30:06.751288 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.751006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls\") pod \"fb53e845-c89d-47f2-852b-7313c92925e3\" (UID: \"fb53e845-c89d-47f2-852b-7313c92925e3\") "
Sep 05 02:30:06.751413 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.751384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb53e845-c89d-47f2-852b-7313c92925e3" (UID: "fb53e845-c89d-47f2-852b-7313c92925e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:06.751476 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.751410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb53e845-c89d-47f2-852b-7313c92925e3" (UID: "fb53e845-c89d-47f2-852b-7313c92925e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:06.751476 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.751419    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb53e845-c89d-47f2-852b-7313c92925e3" (UID: "fb53e845-c89d-47f2-852b-7313c92925e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:30:06.753254 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.753234    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb53e845-c89d-47f2-852b-7313c92925e3" (UID: "fb53e845-c89d-47f2-852b-7313c92925e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:30:06.753324 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.753292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q" (OuterVolumeSpecName: "kube-api-access-dft6q") pod "fb53e845-c89d-47f2-852b-7313c92925e3" (UID: "fb53e845-c89d-47f2-852b-7313c92925e3"). InnerVolumeSpecName "kube-api-access-dft6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:30:06.815489 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815447    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:30:06.815843 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815823    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kube-rbac-proxy"
Sep 05 02:30:06.815950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815845    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kube-rbac-proxy"
Sep 05 02:30:06.815950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815858    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="storage-initializer"
Sep 05 02:30:06.815950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815866    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="storage-initializer"
Sep 05 02:30:06.815950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815903    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container"
Sep 05 02:30:06.815950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815912    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container"
Sep 05 02:30:06.816191 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815976    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kube-rbac-proxy"
Sep 05 02:30:06.816191 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.815988    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" containerName="kserve-container"
Sep 05 02:30:06.819006 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.818980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.834865 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.834834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:30:06.851429 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.851405    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb53e845-c89d-47f2-852b-7313c92925e3-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:06.851429 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.851430    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:06.851597 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.851444    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dft6q\" (UniqueName: \"kubernetes.io/projected/fb53e845-c89d-47f2-852b-7313c92925e3-kube-api-access-dft6q\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:06.851597 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.851457    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb53e845-c89d-47f2-852b-7313c92925e3-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:06.851597 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.851471    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb53e845-c89d-47f2-852b-7313c92925e3-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:06.952380 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.952340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.952380 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.952381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.952614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.952451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.952614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.952480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nzl\" (UniqueName: \"kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.952614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.952570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:06.957639 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.957613    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerID="5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75" exitCode=2
Sep 05 02:30:06.957809 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.957696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerDied","Data":"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"}
Sep 05 02:30:06.959689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.959668    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb53e845-c89d-47f2-852b-7313c92925e3" containerID="3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606" exitCode=0
Sep 05 02:30:06.959802 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.959714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerDied","Data":"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"}
Sep 05 02:30:06.959802 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.959733    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"
Sep 05 02:30:06.959802 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.959784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427" event={"ID":"fb53e845-c89d-47f2-852b-7313c92925e3","Type":"ContainerDied","Data":"a8f854e22caa0709901e5cbf5d839080050201f76f8e3e478f9e3275546f02ce"}
Sep 05 02:30:06.959902 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.959813    2582 scope.go:117] "RemoveContainer" containerID="b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"
Sep 05 02:30:06.968531 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.968515    2582 scope.go:117] "RemoveContainer" containerID="3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"
Sep 05 02:30:06.976235 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.976217    2582 scope.go:117] "RemoveContainer" containerID="c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"
Sep 05 02:30:06.981002 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.980980    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:30:06.984093 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984072    2582 scope.go:117] "RemoveContainer" containerID="b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"
Sep 05 02:30:06.984344 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:06.984327    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466\": container with ID starting with b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466 not found: ID does not exist" containerID="b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"
Sep 05 02:30:06.984404 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984351    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466"} err="failed to get container status \"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466\": rpc error: code = NotFound desc = could not find container \"b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466\": container with ID starting with b15180e929985a72c083501d559b2ccef7cec88a84dd18162fbd0ff9c88b1466 not found: ID does not exist"
Sep 05 02:30:06.984404 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984367    2582 scope.go:117] "RemoveContainer" containerID="3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"
Sep 05 02:30:06.984626 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:06.984610    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606\": container with ID starting with 3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606 not found: ID does not exist" containerID="3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"
Sep 05 02:30:06.984686 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984629    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606"} err="failed to get container status \"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606\": rpc error: code = NotFound desc = could not find container \"3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606\": container with ID starting with 3780a92f863ea5ffa1199391434d8476be17e45d9456bf4ead54c110c2f2a606 not found: ID does not exist"
Sep 05 02:30:06.984686 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984642    2582 scope.go:117] "RemoveContainer" containerID="c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"
Sep 05 02:30:06.984985 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:06.984964    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac\": container with ID starting with c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac not found: ID does not exist" containerID="c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"
Sep 05 02:30:06.985084 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.984992    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac"} err="failed to get container status \"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac\": rpc error: code = NotFound desc = could not find container \"c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac\": container with ID starting with c1c5d8e283f087f1db9fb5c6b28a98c51470536119c9a2c8d93c8cca4249cfac not found: ID does not exist"
Sep 05 02:30:06.986729 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:06.986707    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427"]
Sep 05 02:30:07.053807 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.053706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.053807 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.053753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.053807 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.053799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.054042 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.053822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.054042 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.053848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nzl\" (UniqueName: \"kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.054204 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.054179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.054503 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.054482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.054581 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.054512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.056381 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.056365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.062528 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.062511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nzl\" (UniqueName: \"kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl\") pod \"isvc-canary-rollback-predictor-76998db6fd-p87sf\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.128705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.128672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:07.402157 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.402129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:30:07.404620 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:30:07.404589    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089df05c_6282_4df7_9c70_3d7ae6d5cac2.slice/crio-5a6879ba42348c4a0e84230376b7e9824d124414ce10e520c21bc6676a4dbf2b WatchSource:0}: Error finding container 5a6879ba42348c4a0e84230376b7e9824d124414ce10e520c21bc6676a4dbf2b: Status 404 returned error can't find the container with id 5a6879ba42348c4a0e84230376b7e9824d124414ce10e520c21bc6676a4dbf2b
Sep 05 02:30:07.964839 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.964803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerStarted","Data":"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"}
Sep 05 02:30:07.965295 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:07.964874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerStarted","Data":"5a6879ba42348c4a0e84230376b7e9824d124414ce10e520c21bc6676a4dbf2b"}
Sep 05 02:30:08.182865 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:08.182830    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb53e845-c89d-47f2-852b-7313c92925e3" path="/var/lib/kubelet/pods/fb53e845-c89d-47f2-852b-7313c92925e3/volumes"
Sep 05 02:30:11.657307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.657268    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Sep 05 02:30:11.661755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.661724    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 05 02:30:11.773967 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.773937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:11.820327 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.820301    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:30:11.876820 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.876658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:11.888644 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.888619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert\") pod \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") "
Sep 05 02:30:11.888815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.888654    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls\") pod \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") "
Sep 05 02:30:11.888815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.888677    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location\") pod \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") "
Sep 05 02:30:11.888815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.888703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config\") pod \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") "
Sep 05 02:30:11.888815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.888811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wr8g\" (UniqueName: \"kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g\") pod \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\" (UID: \"d4fe66a8-8b52-446f-9ba1-8dc6e198a353\") "
Sep 05 02:30:11.889061 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.889034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4fe66a8-8b52-446f-9ba1-8dc6e198a353" (UID: "d4fe66a8-8b52-446f-9ba1-8dc6e198a353"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:30:11.889101 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.889068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4fe66a8-8b52-446f-9ba1-8dc6e198a353" (UID: "d4fe66a8-8b52-446f-9ba1-8dc6e198a353"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:11.889132 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.889096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4fe66a8-8b52-446f-9ba1-8dc6e198a353" (UID: "d4fe66a8-8b52-446f-9ba1-8dc6e198a353"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:11.890996 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.890973    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g" (OuterVolumeSpecName: "kube-api-access-8wr8g") pod "d4fe66a8-8b52-446f-9ba1-8dc6e198a353" (UID: "d4fe66a8-8b52-446f-9ba1-8dc6e198a353"). InnerVolumeSpecName "kube-api-access-8wr8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:30:11.891124 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.891095    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4fe66a8-8b52-446f-9ba1-8dc6e198a353" (UID: "d4fe66a8-8b52-446f-9ba1-8dc6e198a353"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:30:11.896210 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896189    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:30:11.896452 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896440    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container"
Sep 05 02:30:11.896492 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896454    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container"
Sep 05 02:30:11.896492 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896465    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kube-rbac-proxy"
Sep 05 02:30:11.896492 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896471    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kube-rbac-proxy"
Sep 05 02:30:11.896492 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896478    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="storage-initializer"
Sep 05 02:30:11.896492 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896483    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="storage-initializer"
Sep 05 02:30:11.896630 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896538    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kube-rbac-proxy"
Sep 05 02:30:11.896630 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.896545    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerName="kserve-container"
Sep 05 02:30:11.899624 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.899609    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.901919 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.901898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3535-predictor-serving-cert\""
Sep 05 02:30:11.902137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.902120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e3535-kube-rbac-proxy-sar-config\""
Sep 05 02:30:11.918705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.918672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:30:11.980814 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.980751    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" containerID="a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89" exitCode=0
Sep 05 02:30:11.980814 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.980805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerDied","Data":"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"}
Sep 05 02:30:11.981041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.980826    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"
Sep 05 02:30:11.981041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.980871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc" event={"ID":"d4fe66a8-8b52-446f-9ba1-8dc6e198a353","Type":"ContainerDied","Data":"dbcee21e70b3b777224fb9b5ba43e7bbf4cd8ae5f1095aed04c44143b8635449"}
Sep 05 02:30:11.981041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.980903    2582 scope.go:117] "RemoveContainer" containerID="5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"
Sep 05 02:30:11.982454 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.982427    2582 generic.go:356] "Generic (PLEG): container finished" podID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerID="1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0" exitCode=0
Sep 05 02:30:11.982557 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.982484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerDied","Data":"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"}
Sep 05 02:30:11.985803 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.985755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:11.989499 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.989603 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85gc\" (UniqueName: \"kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.989603 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.989603 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989692    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wr8g\" (UniqueName: \"kubernetes.io/projected/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-api-access-8wr8g\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989708    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989721    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989733    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:11.989754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.989741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fe66a8-8b52-446f-9ba1-8dc6e198a353-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:30:11.990652 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.990635    2582 scope.go:117] "RemoveContainer" containerID="a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"
Sep 05 02:30:11.999999 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:11.999977    2582 scope.go:117] "RemoveContainer" containerID="e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"
Sep 05 02:30:12.017395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.017371    2582 scope.go:117] "RemoveContainer" containerID="5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"
Sep 05 02:30:12.017807 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:12.017785    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75\": container with ID starting with 5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75 not found: ID does not exist" containerID="5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"
Sep 05 02:30:12.017932 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.017817    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75"} err="failed to get container status \"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75\": rpc error: code = NotFound desc = could not find container \"5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75\": container with ID starting with 5fec0c4cea213d31a139a6a4535bc706f59558cdf7bf1dd82ce137ee40973d75 not found: ID does not exist"
Sep 05 02:30:12.017932 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.017840    2582 scope.go:117] "RemoveContainer" containerID="a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"
Sep 05 02:30:12.018197 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:12.018169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89\": container with ID starting with a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89 not found: ID does not exist" containerID="a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"
Sep 05 02:30:12.018263 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.018224    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89"} err="failed to get container status \"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89\": rpc error: code = NotFound desc = could not find container \"a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89\": container with ID starting with a8410c4d8d6892badfc83d80ef81aaffcd55d3bef6e0fb217a9b7c5fbf12af89 not found: ID does not exist"
Sep 05 02:30:12.018314 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.018242    2582 scope.go:117] "RemoveContainer" containerID="e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"
Sep 05 02:30:12.018620 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:12.018600    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900\": container with ID starting with e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900 not found: ID does not exist" containerID="e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"
Sep 05 02:30:12.018696 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.018628    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900"} err="failed to get container status \"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900\": rpc error: code = NotFound desc = could not find container \"e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900\": container with ID starting with e7c480ca88ba71b2275a582d5b5d60bb70d13da842c815d0bbdc7fdb3fbad900 not found: ID does not exist"
Sep 05 02:30:12.048489 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.048461    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:30:12.072104 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.072073    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc"]
Sep 05 02:30:12.090682 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.090657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.090828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.090709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.090828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.090753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85gc\" (UniqueName: \"kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.090828 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.090804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.090993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.090861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.091052 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:12.090998    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-serving-cert: secret "isvc-sklearn-graph-raw-e3535-predictor-serving-cert" not found
Sep 05 02:30:12.091105 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:30:12.091064    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls podName:5012d0ca-920d-4067-8e96-704da21469f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:30:12.591042073 +0000 UTC m=+746.912814587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls") pod "isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8") : secret "isvc-sklearn-graph-raw-e3535-predictor-serving-cert" not found
Sep 05 02:30:12.091266 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.091214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.091471 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.091453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.091609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.091591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.101713 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.101680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85gc\" (UniqueName: \"kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.123506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.123479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:12.183028 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.182951    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fe66a8-8b52-446f-9ba1-8dc6e198a353" path="/var/lib/kubelet/pods/d4fe66a8-8b52-446f-9ba1-8dc6e198a353/volumes"
Sep 05 02:30:12.218888 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.218845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:12.595705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.595672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.598206 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.598186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.809221 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.809184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:12.989521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.989482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerStarted","Data":"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"}
Sep 05 02:30:12.989672 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.989549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerStarted","Data":"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"}
Sep 05 02:30:12.989857 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.989832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:12.989991 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.989982    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:12.991325 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:12.991297    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:13.012980 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:13.012925    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podStartSLOduration=3.229072782 podStartE2EDuration="7.012909372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.78383659s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:13.011626293 +0000 UTC m=+747.333398811" watchObservedRunningTime="2026-09-05 02:30:13.012909372 +0000 UTC m=+747.334681887"
Sep 05 02:30:13.078895 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:13.078862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:30:13.081729 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:30:13.081700    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5012d0ca_920d_4067_8e96_704da21469f8.slice/crio-95cbdf6432418fe741701657159f3737c875a0cfe1309d40640ffc696df3cfa8 WatchSource:0}: Error finding container 95cbdf6432418fe741701657159f3737c875a0cfe1309d40640ffc696df3cfa8: Status 404 returned error can't find the container with id 95cbdf6432418fe741701657159f3737c875a0cfe1309d40640ffc696df3cfa8
Sep 05 02:30:13.995074 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:13.995038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerStarted","Data":"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"}
Sep 05 02:30:13.995553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:13.995141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerStarted","Data":"95cbdf6432418fe741701657159f3737c875a0cfe1309d40640ffc696df3cfa8"}
Sep 05 02:30:13.995553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:13.995341    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:17.008167 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:17.008132    2582 generic.go:356] "Generic (PLEG): container finished" podID="5012d0ca-920d-4067-8e96-704da21469f8" containerID="f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a" exitCode=0
Sep 05 02:30:17.008554 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:17.008190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerDied","Data":"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"}
Sep 05 02:30:18.014679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:18.014644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerStarted","Data":"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"}
Sep 05 02:30:18.015055 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:18.014709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerStarted","Data":"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"}
Sep 05 02:30:18.015055 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:18.014958    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:18.041824 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:18.041751    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podStartSLOduration=3.331891448 podStartE2EDuration="7.041736184s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.709844736s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:18.040741842 +0000 UTC m=+752.362514373" watchObservedRunningTime="2026-09-05 02:30:18.041736184 +0000 UTC m=+752.363508721"
Sep 05 02:30:19.000676 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:19.000645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:30:19.001348 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:19.001318    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:19.018642 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:19.018617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:19.019896 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:19.019871    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:20.021895 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:20.021859    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:22.962569 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:22.962538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:25.026076 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:25.026046    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:30:25.026526 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:25.026501    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:25.702754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:25.702717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:25.760156 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:25.760122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:29.001526 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:29.001485    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:35.027117 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:35.027080    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:39.001487 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:39.001444    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:45.026815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:45.026750    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:49.001570 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:49.001526    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:30:55.026513 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:55.026472    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:30:59.001538 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:30:59.001493    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:31:05.026681 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:05.026638    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:31:09.002335 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:09.002232    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:31:15.027517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:15.027473    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:31:19.002180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:19.002143    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:31:25.026874 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:25.026835    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:31:29.002207 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:29.002179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:31:29.818062 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:29.818027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:30.179029 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:30.178979    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 05 02:31:36.898274 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:36.898239    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:31:36.898693 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:36.898530    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" containerID="cri-o://9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489" gracePeriod=30
Sep 05 02:31:36.898693 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:36.898553    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kube-rbac-proxy" containerID="cri-o://d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261" gracePeriod=30
Sep 05 02:31:36.927929 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:36.927903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:37.306463 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:37.306377    2582 generic.go:356] "Generic (PLEG): container finished" podID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerID="d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261" exitCode=2
Sep 05 02:31:37.306463 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:37.306452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerDied","Data":"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"}
Sep 05 02:31:38.995713 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:38.995661    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Sep 05 02:31:39.002103 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:39.002068    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 05 02:31:40.183221 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:40.183190    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:31:42.037431 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.037403    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:31:42.038865 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.038833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:42.115902 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.115871    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:31:42.116172 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116156    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="storage-initializer"
Sep 05 02:31:42.116227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116176    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="storage-initializer"
Sep 05 02:31:42.116227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116191    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container"
Sep 05 02:31:42.116227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116196    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container"
Sep 05 02:31:42.116227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116213    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kube-rbac-proxy"
Sep 05 02:31:42.116227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116219    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kube-rbac-proxy"
Sep 05 02:31:42.116392 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116263    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kserve-container"
Sep 05 02:31:42.116392 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.116271    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerName="kube-rbac-proxy"
Sep 05 02:31:42.119322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.119297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.120094 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.120068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:42.121473 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.121454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 05 02:31:42.121637 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.121618    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 05 02:31:42.130996 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.130971    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:31:42.155168 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155132    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert\") pod \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") "
Sep 05 02:31:42.155168 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155167    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68nzl\" (UniqueName: \"kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl\") pod \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") "
Sep 05 02:31:42.155421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls\") pod \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") "
Sep 05 02:31:42.155421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location\") pod \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") "
Sep 05 02:31:42.155421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config\") pod \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\" (UID: \"089df05c-6282-4df7-9c70-3d7ae6d5cac2\") "
Sep 05 02:31:42.155582 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155540    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "089df05c-6282-4df7-9c70-3d7ae6d5cac2" (UID: "089df05c-6282-4df7-9c70-3d7ae6d5cac2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:31:42.155628 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "089df05c-6282-4df7-9c70-3d7ae6d5cac2" (UID: "089df05c-6282-4df7-9c70-3d7ae6d5cac2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:31:42.155714 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155697    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:31:42.155755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155718    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:31:42.155755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.155735    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "089df05c-6282-4df7-9c70-3d7ae6d5cac2" (UID: "089df05c-6282-4df7-9c70-3d7ae6d5cac2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:31:42.157969 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.157949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl" (OuterVolumeSpecName: "kube-api-access-68nzl") pod "089df05c-6282-4df7-9c70-3d7ae6d5cac2" (UID: "089df05c-6282-4df7-9c70-3d7ae6d5cac2"). InnerVolumeSpecName "kube-api-access-68nzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:31:42.157969 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.157956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "089df05c-6282-4df7-9c70-3d7ae6d5cac2" (UID: "089df05c-6282-4df7-9c70-3d7ae6d5cac2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:31:42.256859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.256746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tfbk\" (UniqueName: \"kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.256859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.256805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.256859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.256828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.257141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.256917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.257141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.256979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.257141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.257030    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:31:42.257141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.257044    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nzl\" (UniqueName: \"kubernetes.io/projected/089df05c-6282-4df7-9c70-3d7ae6d5cac2-kube-api-access-68nzl\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:31:42.257141 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.257059    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/089df05c-6282-4df7-9c70-3d7ae6d5cac2-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:31:42.294616 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.294576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:42.329096 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.329058    2582 generic.go:356] "Generic (PLEG): container finished" podID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" containerID="9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489" exitCode=0
Sep 05 02:31:42.329248 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.329120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"
Sep 05 02:31:42.329248 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.329120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerDied","Data":"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"}
Sep 05 02:31:42.329333 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.329257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf" event={"ID":"089df05c-6282-4df7-9c70-3d7ae6d5cac2","Type":"ContainerDied","Data":"5a6879ba42348c4a0e84230376b7e9824d124414ce10e520c21bc6676a4dbf2b"}
Sep 05 02:31:42.329333 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.329295    2582 scope.go:117] "RemoveContainer" containerID="d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"
Sep 05 02:31:42.338327 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.338309    2582 scope.go:117] "RemoveContainer" containerID="9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"
Sep 05 02:31:42.346137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.346115    2582 scope.go:117] "RemoveContainer" containerID="1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"
Sep 05 02:31:42.346753 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.346729    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:31:42.348171 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.348127    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf"]
Sep 05 02:31:42.354128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.354110    2582 scope.go:117] "RemoveContainer" containerID="d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"
Sep 05 02:31:42.354381 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:31:42.354366    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261\": container with ID starting with d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261 not found: ID does not exist" containerID="d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"
Sep 05 02:31:42.354422 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.354390    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261"} err="failed to get container status \"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261\": rpc error: code = NotFound desc = could not find container \"d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261\": container with ID starting with d6d3c5cb05bd01b029e84eea1ce40f2cec2c9daa798e40ce5bc8ef26bfd02261 not found: ID does not exist"
Sep 05 02:31:42.354422 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.354407    2582 scope.go:117] "RemoveContainer" containerID="9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"
Sep 05 02:31:42.354654 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:31:42.354638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489\": container with ID starting with 9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489 not found: ID does not exist" containerID="9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"
Sep 05 02:31:42.354727 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.354665    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489"} err="failed to get container status \"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489\": rpc error: code = NotFound desc = could not find container \"9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489\": container with ID starting with 9f4c2207678ade495e9c41985111f7f3d43c19c63f3750876e8a6b62d7b25489 not found: ID does not exist"
Sep 05 02:31:42.354727 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.354688    2582 scope.go:117] "RemoveContainer" containerID="1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"
Sep 05 02:31:42.355019 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:31:42.355000    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0\": container with ID starting with 1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0 not found: ID does not exist" containerID="1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"
Sep 05 02:31:42.355087 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.355026    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0"} err="failed to get container status \"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0\": rpc error: code = NotFound desc = could not find container \"1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0\": container with ID starting with 1a88b76d7941cb384bc059b10d0bffed94fdd7d99bb29dae25e68a6a32059da0 not found: ID does not exist"
Sep 05 02:31:42.358403 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.358517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tfbk\" (UniqueName: \"kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.358517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.358517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.358666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.358970 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.358951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.359172 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.359152    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.359279 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.359262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.361025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.361010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.367536 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.367509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tfbk\" (UniqueName: \"kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk\") pod \"isvc-canary-stop-predictor-79f67b4d78-2wc45\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.429600 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.429556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:42.710089 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.706552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:31:42.711708 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:31:42.711675    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f4f0a9_9b06_4313_8523_beb895950e3e.slice/crio-9237def304fc66bd45ea0a9656ae90bdfa66aedecff4e65d6a07a501723ff214 WatchSource:0}: Error finding container 9237def304fc66bd45ea0a9656ae90bdfa66aedecff4e65d6a07a501723ff214: Status 404 returned error can't find the container with id 9237def304fc66bd45ea0a9656ae90bdfa66aedecff4e65d6a07a501723ff214
Sep 05 02:31:42.778252 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:42.778218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:43.334309 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:43.334265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerStarted","Data":"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"}
Sep 05 02:31:43.334791 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:43.334337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerStarted","Data":"9237def304fc66bd45ea0a9656ae90bdfa66aedecff4e65d6a07a501723ff214"}
Sep 05 02:31:44.183014 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:44.182984    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089df05c-6282-4df7-9c70-3d7ae6d5cac2" path="/var/lib/kubelet/pods/089df05c-6282-4df7-9c70-3d7ae6d5cac2/volumes"
Sep 05 02:31:47.354460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:47.354426    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerID="05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184" exitCode=0
Sep 05 02:31:47.354870 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:47.354485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerDied","Data":"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"}
Sep 05 02:31:48.360264 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:48.360230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerStarted","Data":"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"}
Sep 05 02:31:48.360788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:48.360292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerStarted","Data":"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"}
Sep 05 02:31:48.360788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:48.360491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:48.379511 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:48.379462    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podStartSLOduration=2.572343216 podStartE2EDuration="6.379446859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807103643s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:48.379170699 +0000 UTC m=+842.700943237" watchObservedRunningTime="2026-09-05 02:31:48.379446859 +0000 UTC m=+842.701219376"
Sep 05 02:31:49.364353 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:49.364319    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:49.365723 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:49.365694    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:31:50.368379 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:50.368336    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:31:51.928067 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:51.928036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:52.255114 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:52.255024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:55.373508 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:55.373480    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:31:55.373993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:31:55.373968    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:01.979019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:01.978970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.057661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.057626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:32:02.057909 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.057883    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container" containerID="cri-o://7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3" gracePeriod=30
Sep 05 02:32:02.057988 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.057924    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kube-rbac-proxy" containerID="cri-o://3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650" gracePeriod=30
Sep 05 02:32:02.091284 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.091249    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.155527 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.155497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.187651 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.187602    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:32:02.190810 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.190792    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.192957 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.192928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bd0a0-kube-rbac-proxy-sar-config\""
Sep 05 02:32:02.193066 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.193006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-serving-cert\""
Sep 05 02:32:02.201122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.201091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:32:02.259039 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.258960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.316677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.316649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.316934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.316685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4cgv\" (UniqueName: \"kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.316934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.316709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.316934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.316795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.316934 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.316834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.320479 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.320455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.414517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.414485    2582 generic.go:356] "Generic (PLEG): container finished" podID="5012d0ca-920d-4067-8e96-704da21469f8" containerID="3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650" exitCode=2
Sep 05 02:32:02.414517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.414521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerDied","Data":"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"}
Sep 05 02:32:02.417846 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.417824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.417916 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.417867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.417916 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.417884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.418019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.417925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.418019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.417952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cgv\" (UniqueName: \"kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.418343 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.418323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.418610 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.418593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.418660 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.418635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.420424 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.420401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.426412 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.426390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cgv\" (UniqueName: \"kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv\") pod \"isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.502083 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.502041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:02.663689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.663655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:02.773623 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:02.773546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:32:02.777686 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:32:02.777653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0e2f08_b0b9_496b_b08f_64a9b133b17c.slice/crio-5d2dc5291cbf34b41a1d3ed8fcfd471bf4c719dfa9ba4cd0d91e565c8c80bc43 WatchSource:0}: Error finding container 5d2dc5291cbf34b41a1d3ed8fcfd471bf4c719dfa9ba4cd0d91e565c8c80bc43: Status 404 returned error can't find the container with id 5d2dc5291cbf34b41a1d3ed8fcfd471bf4c719dfa9ba4cd0d91e565c8c80bc43
Sep 05 02:32:03.420281 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:03.420244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerStarted","Data":"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"}
Sep 05 02:32:03.420787 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:03.420313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerStarted","Data":"5d2dc5291cbf34b41a1d3ed8fcfd471bf4c719dfa9ba4cd0d91e565c8c80bc43"}
Sep 05 02:32:05.022856 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:05.022804    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Sep 05 02:32:05.374619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:05.374575    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:07.002459 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.002425    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:32:07.151506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151471    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location\") pod \"5012d0ca-920d-4067-8e96-704da21469f8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") "
Sep 05 02:32:07.151677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert\") pod \"5012d0ca-920d-4067-8e96-704da21469f8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") "
Sep 05 02:32:07.151677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config\") pod \"5012d0ca-920d-4067-8e96-704da21469f8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") "
Sep 05 02:32:07.151677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p85gc\" (UniqueName: \"kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc\") pod \"5012d0ca-920d-4067-8e96-704da21469f8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") "
Sep 05 02:32:07.151677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151658    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") pod \"5012d0ca-920d-4067-8e96-704da21469f8\" (UID: \"5012d0ca-920d-4067-8e96-704da21469f8\") "
Sep 05 02:32:07.151929 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5012d0ca-920d-4067-8e96-704da21469f8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:07.151929 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151903    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5012d0ca-920d-4067-8e96-704da21469f8-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.152028 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151966    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5012d0ca-920d-4067-8e96-704da21469f8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:07.152028 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.151997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5012d0ca-920d-4067-8e96-704da21469f8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:07.153977 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.153958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5012d0ca-920d-4067-8e96-704da21469f8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:07.154050 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.153992    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc" (OuterVolumeSpecName: "kube-api-access-p85gc") pod "5012d0ca-920d-4067-8e96-704da21469f8" (UID: "5012d0ca-920d-4067-8e96-704da21469f8"). InnerVolumeSpecName "kube-api-access-p85gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:07.253216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.253181    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.253216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.253209    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p85gc\" (UniqueName: \"kubernetes.io/projected/5012d0ca-920d-4067-8e96-704da21469f8-kube-api-access-p85gc\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.253216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.253219    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5012d0ca-920d-4067-8e96-704da21469f8-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.253442 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.253229    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5012d0ca-920d-4067-8e96-704da21469f8-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.437046 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.436958    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerID="05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56" exitCode=0
Sep 05 02:32:07.437194 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.437034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerDied","Data":"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"}
Sep 05 02:32:07.439050 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.439024    2582 generic.go:356] "Generic (PLEG): container finished" podID="5012d0ca-920d-4067-8e96-704da21469f8" containerID="7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3" exitCode=0
Sep 05 02:32:07.439168 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.439091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"
Sep 05 02:32:07.439205 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.439091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerDied","Data":"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"}
Sep 05 02:32:07.439241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.439219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8" event={"ID":"5012d0ca-920d-4067-8e96-704da21469f8","Type":"ContainerDied","Data":"95cbdf6432418fe741701657159f3737c875a0cfe1309d40640ffc696df3cfa8"}
Sep 05 02:32:07.439273 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.439249    2582 scope.go:117] "RemoveContainer" containerID="3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"
Sep 05 02:32:07.450126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.450108    2582 scope.go:117] "RemoveContainer" containerID="7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"
Sep 05 02:32:07.462378 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.462357    2582 scope.go:117] "RemoveContainer" containerID="f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"
Sep 05 02:32:07.469863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.469842    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:32:07.473637 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.473614    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8"]
Sep 05 02:32:07.477800 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.477781    2582 scope.go:117] "RemoveContainer" containerID="3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"
Sep 05 02:32:07.478117 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:32:07.478092    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650\": container with ID starting with 3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650 not found: ID does not exist" containerID="3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"
Sep 05 02:32:07.478185 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.478119    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650"} err="failed to get container status \"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650\": rpc error: code = NotFound desc = could not find container \"3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650\": container with ID starting with 3998cff8a48a2e4ceef2ac8db675119225ff7fbda8b0a10477e849f4444f9650 not found: ID does not exist"
Sep 05 02:32:07.478185 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.478138    2582 scope.go:117] "RemoveContainer" containerID="7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"
Sep 05 02:32:07.478452 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:32:07.478434    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3\": container with ID starting with 7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3 not found: ID does not exist" containerID="7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"
Sep 05 02:32:07.478520 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.478459    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3"} err="failed to get container status \"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3\": rpc error: code = NotFound desc = could not find container \"7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3\": container with ID starting with 7c3ce7de8caf6d39b3b2eeff7da4b17a0f8281cb9331206d8d52098ef6e58fd3 not found: ID does not exist"
Sep 05 02:32:07.478520 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.478475    2582 scope.go:117] "RemoveContainer" containerID="f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"
Sep 05 02:32:07.478749 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:32:07.478726    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a\": container with ID starting with f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a not found: ID does not exist" containerID="f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"
Sep 05 02:32:07.478827 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:07.478776    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a"} err="failed to get container status \"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a\": rpc error: code = NotFound desc = could not find container \"f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a\": container with ID starting with f01ed3cbefa26ebc297e2bf19384a15841947860c1f3d4637e990f253082000a not found: ID does not exist"
Sep 05 02:32:08.182255 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.182221    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5012d0ca-920d-4067-8e96-704da21469f8" path="/var/lib/kubelet/pods/5012d0ca-920d-4067-8e96-704da21469f8/volumes"
Sep 05 02:32:08.445195 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.445100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerStarted","Data":"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"}
Sep 05 02:32:08.445195 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.445171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerStarted","Data":"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"}
Sep 05 02:32:08.445516 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.445489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:08.445701 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.445684    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:08.447024 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.446999    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:08.468068 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:08.468018    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podStartSLOduration=2.508436097 podStartE2EDuration="6.46800131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959565213s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:08.466323708 +0000 UTC m=+862.788096272" watchObservedRunningTime="2026-09-05 02:32:08.46800131 +0000 UTC m=+862.789773827"
Sep 05 02:32:09.450831 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:09.450794    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:14.455017 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:14.454991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:32:14.455497 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:14.455427    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:15.374305 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:15.374255    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:24.455428 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:24.455388    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:25.374689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:25.374646    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:34.456284 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:34.456249    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:35.374307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:35.374273    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:44.456344 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:44.456261    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:45.374606 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:45.374566    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:46.167072 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:46.167045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:32:46.167568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:46.167219    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:32:54.456394 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:54.456353    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:55.374424 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:55.374385    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:32:57.178754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:32:57.178708    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:33:04.455527 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:04.455488    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:33:07.179950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:07.179918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:33:08.679463 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:08.679429    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:33:08.679886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:08.679656    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container" containerID="cri-o://4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25" gracePeriod=30
Sep 05 02:33:08.679886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:08.679686    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kube-rbac-proxy" containerID="cri-o://d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4" gracePeriod=30
Sep 05 02:33:09.671324 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:09.671288    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerID="d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4" exitCode=2
Sep 05 02:33:09.671494 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:09.671358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerDied","Data":"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"}
Sep 05 02:33:10.368803 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:10.368737    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 05 02:33:12.066619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:12.066590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:12.138814 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:12.138783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:13.618413 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.618391    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:33:13.688543 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.688452    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerID="4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25" exitCode=0
Sep 05 02:33:13.688543 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.688519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerDied","Data":"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"}
Sep 05 02:33:13.688543 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.688524    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"
Sep 05 02:33:13.688821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.688579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45" event={"ID":"d6f4f0a9-9b06-4313-8523-beb895950e3e","Type":"ContainerDied","Data":"9237def304fc66bd45ea0a9656ae90bdfa66aedecff4e65d6a07a501723ff214"}
Sep 05 02:33:13.688821 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.688606    2582 scope.go:117] "RemoveContainer" containerID="d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"
Sep 05 02:33:13.696575 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.696557    2582 scope.go:117] "RemoveContainer" containerID="4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"
Sep 05 02:33:13.703824 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.703803    2582 scope.go:117] "RemoveContainer" containerID="05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"
Sep 05 02:33:13.712539 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.712527    2582 scope.go:117] "RemoveContainer" containerID="d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"
Sep 05 02:33:13.713018 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:13.712997    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4\": container with ID starting with d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4 not found: ID does not exist" containerID="d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"
Sep 05 02:33:13.713122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.713025    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4"} err="failed to get container status \"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4\": rpc error: code = NotFound desc = could not find container \"d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4\": container with ID starting with d4adb4f57e19729c5e62afa1b54ff948fe55878ed61ee19931cecface11922b4 not found: ID does not exist"
Sep 05 02:33:13.713122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.713042    2582 scope.go:117] "RemoveContainer" containerID="4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"
Sep 05 02:33:13.713339 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:13.713318    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25\": container with ID starting with 4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25 not found: ID does not exist" containerID="4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"
Sep 05 02:33:13.713405 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.713357    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25"} err="failed to get container status \"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25\": rpc error: code = NotFound desc = could not find container \"4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25\": container with ID starting with 4406f8fdddd2e2fad5a8e519d42285a9143cd84d6a300c7ea42718e35d46fb25 not found: ID does not exist"
Sep 05 02:33:13.713405 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.713380    2582 scope.go:117] "RemoveContainer" containerID="05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"
Sep 05 02:33:13.713649 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:13.713635    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184\": container with ID starting with 05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184 not found: ID does not exist" containerID="05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"
Sep 05 02:33:13.713690 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.713654    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184"} err="failed to get container status \"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184\": rpc error: code = NotFound desc = could not find container \"05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184\": container with ID starting with 05f1421775f1e6cf8fd5494f82548a98ff41b9d1ef75acafced3ac10e36ca184 not found: ID does not exist"
Sep 05 02:33:13.717929 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.717911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location\") pod \"d6f4f0a9-9b06-4313-8523-beb895950e3e\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") "
Sep 05 02:33:13.717983 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.717972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tfbk\" (UniqueName: \"kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk\") pod \"d6f4f0a9-9b06-4313-8523-beb895950e3e\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") "
Sep 05 02:33:13.718025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert\") pod \"d6f4f0a9-9b06-4313-8523-beb895950e3e\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") "
Sep 05 02:33:13.718075 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config\") pod \"d6f4f0a9-9b06-4313-8523-beb895950e3e\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") "
Sep 05 02:33:13.718123 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718094    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls\") pod \"d6f4f0a9-9b06-4313-8523-beb895950e3e\" (UID: \"d6f4f0a9-9b06-4313-8523-beb895950e3e\") "
Sep 05 02:33:13.718211 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718191    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6f4f0a9-9b06-4313-8523-beb895950e3e" (UID: "d6f4f0a9-9b06-4313-8523-beb895950e3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:13.718342 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718328    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f4f0a9-9b06-4313-8523-beb895950e3e-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:13.718423 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6f4f0a9-9b06-4313-8523-beb895950e3e" (UID: "d6f4f0a9-9b06-4313-8523-beb895950e3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:13.718482 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.718446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6f4f0a9-9b06-4313-8523-beb895950e3e" (UID: "d6f4f0a9-9b06-4313-8523-beb895950e3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:13.720045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.720021    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk" (OuterVolumeSpecName: "kube-api-access-9tfbk") pod "d6f4f0a9-9b06-4313-8523-beb895950e3e" (UID: "d6f4f0a9-9b06-4313-8523-beb895950e3e"). InnerVolumeSpecName "kube-api-access-9tfbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:13.720171 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.720144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6f4f0a9-9b06-4313-8523-beb895950e3e" (UID: "d6f4f0a9-9b06-4313-8523-beb895950e3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:13.819679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.819643    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:13.819679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.819672    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f4f0a9-9b06-4313-8523-beb895950e3e-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:13.819679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.819684    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tfbk\" (UniqueName: \"kubernetes.io/projected/d6f4f0a9-9b06-4313-8523-beb895950e3e-kube-api-access-9tfbk\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:13.819910 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:13.819693    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f4f0a9-9b06-4313-8523-beb895950e3e-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:14.012183 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:14.012157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:33:14.018730 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:14.018701    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45"]
Sep 05 02:33:14.182522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:14.182480    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" path="/var/lib/kubelet/pods/d6f4f0a9-9b06-4313-8523-beb895950e3e/volumes"
Sep 05 02:33:14.456425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:14.456389    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:33:22.312609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:22.312567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:23.675813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:23.675757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:24.456890 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:24.456864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:33:36.465899 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:36.465862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:42.007403 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:42.007361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:42.067990 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:42.067958    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:42.182715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:42.182667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:42.553540 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:42.553503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:52.260345 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.260300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:52.342395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.342361    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:33:52.342663 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.342632    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" containerID="cri-o://d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d" gracePeriod=30
Sep 05 02:33:52.342757 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.342680    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kube-rbac-proxy" containerID="cri-o://eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f" gracePeriod=30
Sep 05 02:33:52.367127 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.367092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:52.453179 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.453142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:52.506268 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.506241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:52.829826 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.829788    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerID="eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f" exitCode=2
Sep 05 02:33:52.829996 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.829792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerDied","Data":"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"}
Sep 05 02:33:52.960746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:52.960715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:54.450914 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:54.450881    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 05 02:33:54.455870 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:54.455842    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:33:57.083135 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.083112    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:33:57.142100 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls\") pod \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") "
Sep 05 02:33:57.142254 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location\") pod \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") "
Sep 05 02:33:57.142254 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142132    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4cgv\" (UniqueName: \"kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv\") pod \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") "
Sep 05 02:33:57.142254 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142153    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert\") pod \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") "
Sep 05 02:33:57.142254 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config\") pod \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\" (UID: \"2e0e2f08-b0b9-496b-b08f-64a9b133b17c\") "
Sep 05 02:33:57.142562 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e0e2f08-b0b9-496b-b08f-64a9b133b17c" (UID: "2e0e2f08-b0b9-496b-b08f-64a9b133b17c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:57.142689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142571    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e0e2f08-b0b9-496b-b08f-64a9b133b17c" (UID: "2e0e2f08-b0b9-496b-b08f-64a9b133b17c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:57.142689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.142612    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e0e2f08-b0b9-496b-b08f-64a9b133b17c" (UID: "2e0e2f08-b0b9-496b-b08f-64a9b133b17c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:57.144316 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.144298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e0e2f08-b0b9-496b-b08f-64a9b133b17c" (UID: "2e0e2f08-b0b9-496b-b08f-64a9b133b17c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:57.144473 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.144450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv" (OuterVolumeSpecName: "kube-api-access-b4cgv") pod "2e0e2f08-b0b9-496b-b08f-64a9b133b17c" (UID: "2e0e2f08-b0b9-496b-b08f-64a9b133b17c"). InnerVolumeSpecName "kube-api-access-b4cgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:57.242844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.242748    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:57.242844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.242793    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:57.242844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.242804    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:57.242844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.242812    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4cgv\" (UniqueName: \"kubernetes.io/projected/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-kube-api-access-b4cgv\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:57.242844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.242821    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e0e2f08-b0b9-496b-b08f-64a9b133b17c-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:33:57.848972 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.848936    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerID="d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d" exitCode=0
Sep 05 02:33:57.849150 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.848997    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"
Sep 05 02:33:57.849150 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.849016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerDied","Data":"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"}
Sep 05 02:33:57.849150 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.849084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n" event={"ID":"2e0e2f08-b0b9-496b-b08f-64a9b133b17c","Type":"ContainerDied","Data":"5d2dc5291cbf34b41a1d3ed8fcfd471bf4c719dfa9ba4cd0d91e565c8c80bc43"}
Sep 05 02:33:57.849150 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.849111    2582 scope.go:117] "RemoveContainer" containerID="eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"
Sep 05 02:33:57.857651 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.857628    2582 scope.go:117] "RemoveContainer" containerID="d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"
Sep 05 02:33:57.865090 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.865074    2582 scope.go:117] "RemoveContainer" containerID="05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"
Sep 05 02:33:57.871840 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.871821    2582 scope.go:117] "RemoveContainer" containerID="eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"
Sep 05 02:33:57.872116 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:57.872096    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f\": container with ID starting with eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f not found: ID does not exist" containerID="eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"
Sep 05 02:33:57.872198 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.872126    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f"} err="failed to get container status \"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f\": rpc error: code = NotFound desc = could not find container \"eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f\": container with ID starting with eb4cb3bcff13dd7ad41df9dc43ba7480932b312958aaefe9b0be9e755e73d39f not found: ID does not exist"
Sep 05 02:33:57.872198 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.872142    2582 scope.go:117] "RemoveContainer" containerID="d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"
Sep 05 02:33:57.872408 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:57.872390    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d\": container with ID starting with d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d not found: ID does not exist" containerID="d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"
Sep 05 02:33:57.872446 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.872414    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d"} err="failed to get container status \"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d\": rpc error: code = NotFound desc = could not find container \"d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d\": container with ID starting with d221d95232d76c9303f349c8c05ea5ceaa6c37aa8abb99afa3d489b47b2b925d not found: ID does not exist"
Sep 05 02:33:57.872446 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.872431    2582 scope.go:117] "RemoveContainer" containerID="05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"
Sep 05 02:33:57.872696 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:33:57.872676    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56\": container with ID starting with 05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56 not found: ID does not exist" containerID="05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"
Sep 05 02:33:57.872755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.872702    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56"} err="failed to get container status \"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56\": rpc error: code = NotFound desc = could not find container \"05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56\": container with ID starting with 05de75d110bfccb4188e4b7da4db67e42abf3192c84a1865e17beb171f37df56 not found: ID does not exist"
Sep 05 02:33:57.876312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.876292    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:33:57.880149 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:57.880127    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n"]
Sep 05 02:33:58.182524 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:33:58.182444    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" path="/var/lib/kubelet/pods/2e0e2f08-b0b9-496b-b08f-64a9b133b17c/volumes"
Sep 05 02:34:12.433662 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.433627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:12.470053 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470022    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:34:12.470273 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470261    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470275    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470285    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470291    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470298    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470303    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470311    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470316    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470323    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="storage-initializer"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="storage-initializer"
Sep 05 02:34:12.470330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470334    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470341    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470350    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470355    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470362    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="storage-initializer"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470367    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="storage-initializer"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470376    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="storage-initializer"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470380    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="storage-initializer"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470419    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470427    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kserve-container"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470433    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0e2f08-b0b9-496b-b08f-64a9b133b17c" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470439    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kserve-container"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470444    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5012d0ca-920d-4067-8e96-704da21469f8" containerName="kube-rbac-proxy"
Sep 05 02:34:12.470666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.470450    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f4f0a9-9b06-4313-8523-beb895950e3e" containerName="kserve-container"
Sep 05 02:34:12.473473 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.473453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.475919 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.475895    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:34:12.476040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.475920    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:34:12.476040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.475924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:34:12.476040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.475942    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d98d3-kube-rbac-proxy-sar-config\""
Sep 05 02:34:12.476040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.475980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:34:12.476250 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.476212    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:34:12.476307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.476262    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d98d3-predictor-serving-cert\""
Sep 05 02:34:12.483093 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.483074    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:34:12.552501 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.552471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9rb\" (UniqueName: \"kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.552501 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.552511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.552726 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.552531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.552726 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.552553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.552726 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.552574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653167 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9rb\" (UniqueName: \"kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653519 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.653570 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:34:12.653545    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-serving-cert: secret "isvc-logger-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:34:12.653626 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:34:12.653613    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls podName:babb3641-0bd8-41f2-8fc9-8b1cee46fd4c nodeName:}" failed. No retries permitted until 2026-09-05 02:34:13.153592897 +0000 UTC m=+987.475365402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls") pod "isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c") : secret "isvc-logger-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:34:12.653669 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.653550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.654052 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.654032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.654096 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.654064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:12.661718 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:12.661699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9rb\" (UniqueName: \"kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:13.157278 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:13.157228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:13.157479 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:34:13.157389    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-serving-cert: secret "isvc-logger-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:34:13.157479 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:34:13.157465    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls podName:babb3641-0bd8-41f2-8fc9-8b1cee46fd4c nodeName:}" failed. No retries permitted until 2026-09-05 02:34:14.157448435 +0000 UTC m=+988.479220930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls") pod "isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c") : secret "isvc-logger-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:34:13.310007 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:13.309972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:14.164011 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.163965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:14.166434 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.166414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:14.284538 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.284490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:14.567598 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.567524    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:34:14.570416 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:34:14.570381    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabb3641_0bd8_41f2_8fc9_8b1cee46fd4c.slice/crio-e82c9e477f6fc72da6c44149cba848b943cf97574a048a452018901b786082dc WatchSource:0}: Error finding container e82c9e477f6fc72da6c44149cba848b943cf97574a048a452018901b786082dc: Status 404 returned error can't find the container with id e82c9e477f6fc72da6c44149cba848b943cf97574a048a452018901b786082dc
Sep 05 02:34:14.910204 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.910170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerStarted","Data":"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"}
Sep 05 02:34:14.910411 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:14.910232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerStarted","Data":"e82c9e477f6fc72da6c44149cba848b943cf97574a048a452018901b786082dc"}
Sep 05 02:34:18.926047 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:18.926014    2582 generic.go:356] "Generic (PLEG): container finished" podID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerID="9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324" exitCode=0
Sep 05 02:34:18.926425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:18.926089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerDied","Data":"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"}
Sep 05 02:34:19.048497 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:19.048477    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:34:19.933954 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:19.933910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerStarted","Data":"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"}
Sep 05 02:34:19.934415 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:19.934039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerStarted","Data":"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"}
Sep 05 02:34:21.943987 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:21.943950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerStarted","Data":"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"}
Sep 05 02:34:21.944395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:21.944117    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:21.966829 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:21.966777    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podStartSLOduration=3.323880726 podStartE2EDuration="9.966746194s" totalImagesPullingTime="2.808815589s" totalInitContainerRuntime="3.834049879s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:34:21.965256857 +0000 UTC m=+996.287029399" watchObservedRunningTime="2026-09-05 02:34:21.966746194 +0000 UTC m=+996.288518709"
Sep 05 02:34:22.947536 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:22.947503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:22.948006 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:22.947568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:22.948949 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:22.948914    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:22.949648 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:22.949627    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:23.951441 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:23.951399    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:23.951935 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:23.951917    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:28.956783 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:28.956733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:34:28.957418 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:28.957362    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:28.957651 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:28.957622    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:38.957604 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:38.957564    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:38.958124 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:38.958097    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:48.958004 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:48.957958    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:48.958493 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:48.958424    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:34:58.958108 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:58.958063    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:34:58.958578 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:34:58.958548    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:07.454282 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.454242    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:07.483817 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.483786    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:07.488811 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.488789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.491078 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.491046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b3e077-dockercfg-2rl6k\""
Sep 05 02:35:07.491361 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.491339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b3e077\""
Sep 05 02:35:07.491454 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.491358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b3e077-predictor-serving-cert\""
Sep 05 02:35:07.495953 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.495932    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b3e077-kube-rbac-proxy-sar-config\""
Sep 05 02:35:07.499395 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.499361    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:07.667583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.667545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.667813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.667596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.667813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.667646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7zp5\" (UniqueName: \"kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.667813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.667687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.667813 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.667748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768408 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768408 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768408 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zp5\" (UniqueName: \"kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768745 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768745 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.768745 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:35:07.768562    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-serving-cert: secret "isvc-secondary-b3e077-predictor-serving-cert" not found
Sep 05 02:35:07.768745 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:35:07.768630    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls podName:2b52d1d0-69cb-4f87-9d24-d68b8a90dcef nodeName:}" failed. No retries permitted until 2026-09-05 02:35:08.268610058 +0000 UTC m=+1042.590382572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls") pod "isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef") : secret "isvc-secondary-b3e077-predictor-serving-cert" not found
Sep 05 02:35:07.768953 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.768815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.769446 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.769417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.769547 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.769470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:07.777456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:07.777429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7zp5\" (UniqueName: \"kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:08.109138 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.109111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:08.273473 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.273436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:08.276045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.276017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") pod \"isvc-secondary-b3e077-predictor-556784fdbf-vrbrt\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") " pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:08.408730 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.408639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:08.679579 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.679487    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:08.683949 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:35:08.683915    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b52d1d0_69cb_4f87_9d24_d68b8a90dcef.slice/crio-953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8 WatchSource:0}: Error finding container 953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8: Status 404 returned error can't find the container with id 953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8
Sep 05 02:35:08.958153 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.958060    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:35:08.958611 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:08.958591    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:09.113158 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:09.113121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerStarted","Data":"61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"}
Sep 05 02:35:09.113325 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:09.113178    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerStarted","Data":"953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8"}
Sep 05 02:35:14.132859 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:14.132828    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/0.log"
Sep 05 02:35:14.133348 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:14.132870    2582 generic.go:356] "Generic (PLEG): container finished" podID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerID="61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7" exitCode=1
Sep 05 02:35:14.133348 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:14.132943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerDied","Data":"61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"}
Sep 05 02:35:15.138820 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:15.138791    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/0.log"
Sep 05 02:35:15.139246 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:15.138845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerStarted","Data":"d5543cd603e3d0921ca9fff8aa79c749ee3814aaf2068ae096ed9b909447c1a0"}
Sep 05 02:35:18.958158 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:18.958107    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:35:18.958575 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:18.958553    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:19.155448 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.155420    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/1.log"
Sep 05 02:35:19.155858 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.155842    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/0.log"
Sep 05 02:35:19.155927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.155876    2582 generic.go:356] "Generic (PLEG): container finished" podID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerID="d5543cd603e3d0921ca9fff8aa79c749ee3814aaf2068ae096ed9b909447c1a0" exitCode=1
Sep 05 02:35:19.155927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.155913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerDied","Data":"d5543cd603e3d0921ca9fff8aa79c749ee3814aaf2068ae096ed9b909447c1a0"}
Sep 05 02:35:19.155996 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.155967    2582 scope.go:117] "RemoveContainer" containerID="61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"
Sep 05 02:35:19.156335 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.156311    2582 scope.go:117] "RemoveContainer" containerID="61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"
Sep 05 02:35:19.166883 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:35:19.166861    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_kserve-ci-e2e-test(2b52d1d0-69cb-4f87-9d24-d68b8a90dcef)\"" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"
Sep 05 02:35:19.167092 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:35:19.167066    2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_kserve-ci-e2e-test_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef_0 in pod sandbox 953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8 from index: no such id: '61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7'" containerID="61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"
Sep 05 02:35:19.167145 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:19.167107    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_kserve-ci-e2e-test_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef_0 in pod sandbox 953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8 from index: no such id: '61591c1ee6e456a96c6ca1c4cf064c3ee74d8bfbbc11dbf8b480f2d2d79c08d7'"
Sep 05 02:35:20.160536 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:20.160508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/1.log"
Sep 05 02:35:25.523092 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.523058    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:25.550779 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.550732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:25.649002 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.648962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:25.653993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.653974    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/1.log"
Sep 05 02:35:25.657736 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.657722    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:25.811737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.811640    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7zp5\" (UniqueName: \"kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5\") pod \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") "
Sep 05 02:35:25.811737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.811689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert\") pod \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") "
Sep 05 02:35:25.811737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.811716    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config\") pod \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") "
Sep 05 02:35:25.812036 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.811756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location\") pod \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") "
Sep 05 02:35:25.812036 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.811807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") pod \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\" (UID: \"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef\") "
Sep 05 02:35:25.812125 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.812094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:25.812180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.812149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:25.812180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.812152    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:25.814067 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.814042    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5" (OuterVolumeSpecName: "kube-api-access-l7zp5") pod "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"). InnerVolumeSpecName "kube-api-access-l7zp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:25.814192 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.814040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" (UID: "2b52d1d0-69cb-4f87-9d24-d68b8a90dcef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:25.913064 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.913019    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7zp5\" (UniqueName: \"kubernetes.io/projected/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-api-access-l7zp5\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:35:25.913064 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.913057    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:35:25.913064 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.913069    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:35:25.913303 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.913077    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:35:25.913303 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:25.913087    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:35:26.183467 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.183440    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/storage-initializer/1.log"
Sep 05 02:35:26.183644 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.183501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt" event={"ID":"2b52d1d0-69cb-4f87-9d24-d68b8a90dcef","Type":"ContainerDied","Data":"953b1c53f0adb56039f7cb13aedae1d39a47997e5a01400a3ba5604343fd4ac8"}
Sep 05 02:35:26.183644 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.183527    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"
Sep 05 02:35:26.183644 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.183549    2582 scope.go:117] "RemoveContainer" containerID="d5543cd603e3d0921ca9fff8aa79c749ee3814aaf2068ae096ed9b909447c1a0"
Sep 05 02:35:26.220372 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.220340    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:26.223226 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:26.223193    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt"]
Sep 05 02:35:28.182595 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:28.182562    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" path="/var/lib/kubelet/pods/2b52d1d0-69cb-4f87-9d24-d68b8a90dcef/volumes"
Sep 05 02:35:28.957568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:28.957532    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:35:28.957977 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:28.957954    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:30.698579 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:30.698541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:31.186661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:31.186624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:38.958030 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:38.957946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:35:38.958517 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:38.958206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:35:40.732087 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:40.732055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:45.978019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:45.977983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:46.322614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:46.322576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:47.605322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.605268    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:35:47.606004 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.605544    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" containerID="cri-o://ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307" gracePeriod=30
Sep 05 02:35:47.606004 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.605557    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" containerID="cri-o://fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040" gracePeriod=30
Sep 05 02:35:47.606004 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.605608    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" containerID="cri-o://e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d" gracePeriod=30
Sep 05 02:35:47.631431 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.631404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:47.734196 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.734165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:47.740843 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.740820    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:35:47.741158 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741142    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.741233 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741161    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.741298 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741236    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.741298 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741248    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.741398 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741320    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.741398 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.741332    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b52d1d0-69cb-4f87-9d24-d68b8a90dcef" containerName="storage-initializer"
Sep 05 02:35:47.744354 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.744332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.746672 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.746641    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a62a7-kube-rbac-proxy-sar-config\""
Sep 05 02:35:47.746672 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.746662    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a62a7-predictor-serving-cert\""
Sep 05 02:35:47.754791 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.754740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:35:47.771869 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.771806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.772040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.772015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.772159 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.772134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzzk\" (UniqueName: \"kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.772266 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.772247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.772311 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.772303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.831495 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.831458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:47.872811 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.872723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.872811 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.872756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.872811 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.872801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.873068 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.872821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzzk\" (UniqueName: \"kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.873068 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.872854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.873301 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.873281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.873587 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.873562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.873653 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.873586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.875293 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.875275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.881982 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.881960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzzk\" (UniqueName: \"kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk\") pod \"isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:47.910810 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:47.910751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:48.055458 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.055427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:48.268130 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.268038    2582 generic.go:356] "Generic (PLEG): container finished" podID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerID="e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d" exitCode=2
Sep 05 02:35:48.268130 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.268112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerDied","Data":"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"}
Sep 05 02:35:48.331958 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.331928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:35:48.334850 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:35:48.334826    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cfdf36_52ca_4fd2_be92_8c5e401170a2.slice/crio-df1fe2884b56b980520f6ae35d75caf2e520eab83a25ea1f4c095ad586d646e6 WatchSource:0}: Error finding container df1fe2884b56b980520f6ae35d75caf2e520eab83a25ea1f4c095ad586d646e6: Status 404 returned error can't find the container with id df1fe2884b56b980520f6ae35d75caf2e520eab83a25ea1f4c095ad586d646e6
Sep 05 02:35:48.952442 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.952397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:35:48.957866 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.957832    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:35:48.958217 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:48.958196    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:49.273715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:49.273635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerStarted","Data":"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"}
Sep 05 02:35:49.273715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:49.273688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerStarted","Data":"df1fe2884b56b980520f6ae35d75caf2e520eab83a25ea1f4c095ad586d646e6"}
Sep 05 02:35:52.286970 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:52.286937    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerID="1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b" exitCode=0
Sep 05 02:35:52.287386 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:52.287015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerDied","Data":"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"}
Sep 05 02:35:53.292755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.292719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerStarted","Data":"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"}
Sep 05 02:35:53.293220 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.292821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerStarted","Data":"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"}
Sep 05 02:35:53.293220 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.293134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:53.293220 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.293179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:53.295001 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.294974    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:35:53.295542 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.295521    2582 generic.go:356] "Generic (PLEG): container finished" podID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerID="ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307" exitCode=0
Sep 05 02:35:53.295614 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.295552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerDied","Data":"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"}
Sep 05 02:35:53.315372 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.315322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podStartSLOduration=2.654059503 podStartE2EDuration="6.315306008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.661246505s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:53.31402455 +0000 UTC m=+1087.635797081" watchObservedRunningTime="2026-09-05 02:35:53.315306008 +0000 UTC m=+1087.637078525"
Sep 05 02:35:53.952049 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:53.951998    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:35:54.299490 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:54.299393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:35:58.951609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:58.951567    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:35:58.952002 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:58.951687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:35:58.957305 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:58.957248    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:35:58.957648 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:58.957625    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:35:59.304866 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:59.304786    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:35:59.305241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:35:59.305212    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:03.951950 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:03.951906    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:36:08.952124 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:08.952079    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:36:08.958120 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:08.958086    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:36:08.958275 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:08.958262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:36:08.958375 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:08.958350    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:36:08.958456 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:08.958445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:36:09.306275 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:09.306186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:13.952224 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:13.952187    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:36:17.770385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.770354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:36:17.892556 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config\") pod \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") "
Sep 05 02:36:17.892556 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd9rb\" (UniqueName: \"kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb\") pod \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") "
Sep 05 02:36:17.892556 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location\") pod \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") "
Sep 05 02:36:17.892885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert\") pod \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") "
Sep 05 02:36:17.892885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") pod \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\" (UID: \"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c\") "
Sep 05 02:36:17.892991 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:36:17.893037 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.892988    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:36:17.893037 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.893009    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:17.894944 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.894906    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:36:17.895062 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.894953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb" (OuterVolumeSpecName: "kube-api-access-kd9rb") pod "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" (UID: "babb3641-0bd8-41f2-8fc9-8b1cee46fd4c"). InnerVolumeSpecName "kube-api-access-kd9rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:17.994005 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.993971    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:36:17.994005 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.994003    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9rb\" (UniqueName: \"kubernetes.io/projected/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kube-api-access-kd9rb\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:36:17.994005 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.994012    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:36:17.994243 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.994022    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:36:17.994243 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:17.994031    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:36:18.387113 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.387083    2582 generic.go:356] "Generic (PLEG): container finished" podID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerID="fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040" exitCode=137
Sep 05 02:36:18.387261 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.387160    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"
Sep 05 02:36:18.387261 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.387171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerDied","Data":"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"}
Sep 05 02:36:18.387261 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.387251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w" event={"ID":"babb3641-0bd8-41f2-8fc9-8b1cee46fd4c","Type":"ContainerDied","Data":"e82c9e477f6fc72da6c44149cba848b943cf97574a048a452018901b786082dc"}
Sep 05 02:36:18.387389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.387306    2582 scope.go:117] "RemoveContainer" containerID="fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"
Sep 05 02:36:18.395226 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.395206    2582 scope.go:117] "RemoveContainer" containerID="e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"
Sep 05 02:36:18.402735 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.402716    2582 scope.go:117] "RemoveContainer" containerID="ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"
Sep 05 02:36:18.404927 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.404906    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:36:18.408699 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.408678    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w"]
Sep 05 02:36:18.411488 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.411471    2582 scope.go:117] "RemoveContainer" containerID="9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"
Sep 05 02:36:18.418751 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.418736    2582 scope.go:117] "RemoveContainer" containerID="fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"
Sep 05 02:36:18.419016 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:36:18.418991    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040\": container with ID starting with fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040 not found: ID does not exist" containerID="fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"
Sep 05 02:36:18.419071 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419024    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040"} err="failed to get container status \"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040\": rpc error: code = NotFound desc = could not find container \"fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040\": container with ID starting with fa228aa1fad0a52661a7dcbbf868007a023cd1876dba792fdf90b2ea63070040 not found: ID does not exist"
Sep 05 02:36:18.419071 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419042    2582 scope.go:117] "RemoveContainer" containerID="e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"
Sep 05 02:36:18.419290 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:36:18.419275    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d\": container with ID starting with e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d not found: ID does not exist" containerID="e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"
Sep 05 02:36:18.419337 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419296    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d"} err="failed to get container status \"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d\": rpc error: code = NotFound desc = could not find container \"e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d\": container with ID starting with e6a6584df6fec4efc6be3b491cf7df98b4947fceaaa60de15a8d64c18424df6d not found: ID does not exist"
Sep 05 02:36:18.419337 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419309    2582 scope.go:117] "RemoveContainer" containerID="ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"
Sep 05 02:36:18.419507 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:36:18.419492    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307\": container with ID starting with ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307 not found: ID does not exist" containerID="ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"
Sep 05 02:36:18.419563 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419510    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307"} err="failed to get container status \"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307\": rpc error: code = NotFound desc = could not find container \"ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307\": container with ID starting with ae45607250c1659f434215a431c50e2ccde5be6bb11f0aff141a34e010d3d307 not found: ID does not exist"
Sep 05 02:36:18.419563 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419522    2582 scope.go:117] "RemoveContainer" containerID="9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"
Sep 05 02:36:18.419699 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:36:18.419685    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324\": container with ID starting with 9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324 not found: ID does not exist" containerID="9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"
Sep 05 02:36:18.419740 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:18.419705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324"} err="failed to get container status \"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324\": rpc error: code = NotFound desc = could not find container \"9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324\": container with ID starting with 9d663ab5d6d235efaed6e064d6a6cd6751b8e79d9003a46a0dd1936fb683f324 not found: ID does not exist"
Sep 05 02:36:19.306057 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:19.306007    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:20.182858 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:20.182826    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" path="/var/lib/kubelet/pods/babb3641-0bd8-41f2-8fc9-8b1cee46fd4c/volumes"
Sep 05 02:36:29.305362 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:29.305323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:39.305244 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:39.305202    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:49.306143 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:49.306102    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:36:52.387343 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:52.387305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:59.305559 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:36:59.305523    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:09.305631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:09.305547    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:11.178598 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:11.178559    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:16.040045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:16.040011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:21.163118 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:21.163087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:21.179279 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:21.179240    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:21.351240 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:21.351201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:31.179367 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:31.179320    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:41.178723 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:41.178673    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:37:46.190868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:46.190840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:37:46.191844 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:46.191825    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:37:51.178583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:37:51.178538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:38:01.179409 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:01.179360    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:38:11.179863 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:11.179822    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:38:21.179452 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:21.179408    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:38:22.372620 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:22.372583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:22.471123 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:22.471090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:27.374173 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:27.374144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:29.040725 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:29.040686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:29.828392 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:29.828356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:31.179424 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:31.179391    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:38:37.837721 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:37.837646    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:38:37.838154 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:37.837897    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" containerID="cri-o://cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5" gracePeriod=30
Sep 05 02:38:37.838154 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:37.837928    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy" containerID="cri-o://9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8" gracePeriod=30
Sep 05 02:38:37.864309 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:37.864279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:38.883728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:38.883698    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerID="9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8" exitCode=2
Sep 05 02:38:38.884113 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:38.883793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerDied","Data":"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"}
Sep 05 02:38:39.300460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:39.300360    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 05 02:38:41.178603 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:41.178555    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:38:43.054156 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.054115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:43.153718 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.153686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:43.252704 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252677    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:38:43.252953 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252940    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252954    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252969    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252975    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252982    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="storage-initializer"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252988    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="storage-initializer"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.252994    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy"
Sep 05 02:38:43.252997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.253000    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy"
Sep 05 02:38:43.253272 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.253043    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="agent"
Sep 05 02:38:43.253272 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.253049    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kube-rbac-proxy"
Sep 05 02:38:43.253272 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.253056    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="babb3641-0bd8-41f2-8fc9-8b1cee46fd4c" containerName="kserve-container"
Sep 05 02:38:43.256084 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.256068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.259401 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.259381    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 05 02:38:43.260698 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.260681    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 05 02:38:43.275410 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.275389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:38:43.424265 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.424212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.424265 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.424262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9cp\" (UniqueName: \"kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.424506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.424296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.424506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.424370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.424506 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.424420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.506468 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.506431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:43.525563 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.525529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.525728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.525570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.525728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.525608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.525728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.525659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.525728 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.525688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9cp\" (UniqueName: \"kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.526120 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.526098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.526387 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.526366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.526431 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.526371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.528241 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.528223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.534886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.534861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9cp\" (UniqueName: \"kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp\") pod \"isvc-canary-create-predictor-54dc6cb966-c9kz2\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.566030 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.565989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:43.604925 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.604897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:43.843777 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.843731    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:38:43.846788 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:38:43.846743    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5ded96_f97e_4b49_8c32_77b7b1c20aa6.slice/crio-14fe04ab17980a3938ef3d830ae3a9f14c7011137c923e944c0081bf0a1d60ff WatchSource:0}: Error finding container 14fe04ab17980a3938ef3d830ae3a9f14c7011137c923e944c0081bf0a1d60ff: Status 404 returned error can't find the container with id 14fe04ab17980a3938ef3d830ae3a9f14c7011137c923e944c0081bf0a1d60ff
Sep 05 02:38:43.908333 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:43.908301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerStarted","Data":"14fe04ab17980a3938ef3d830ae3a9f14c7011137c923e944c0081bf0a1d60ff"}
Sep 05 02:38:44.299819 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:44.299708    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 05 02:38:44.913305 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:44.913271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerStarted","Data":"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"}
Sep 05 02:38:47.878353 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.878327    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:38:47.925551 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.925465    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerID="1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052" exitCode=0
Sep 05 02:38:47.925715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.925542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerDied","Data":"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"}
Sep 05 02:38:47.927568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.927536    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerID="cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5" exitCode=0
Sep 05 02:38:47.927679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.927585    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"
Sep 05 02:38:47.927679 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.927579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerDied","Data":"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"}
Sep 05 02:38:47.927784 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.927703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4" event={"ID":"e2cfdf36-52ca-4fd2-be92-8c5e401170a2","Type":"ContainerDied","Data":"df1fe2884b56b980520f6ae35d75caf2e520eab83a25ea1f4c095ad586d646e6"}
Sep 05 02:38:47.927784 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.927729    2582 scope.go:117] "RemoveContainer" containerID="9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"
Sep 05 02:38:47.937672 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.937653    2582 scope.go:117] "RemoveContainer" containerID="cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"
Sep 05 02:38:47.952912 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.951279    2582 scope.go:117] "RemoveContainer" containerID="1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"
Sep 05 02:38:47.959886 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.959855    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location\") pod \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") "
Sep 05 02:38:47.960022 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.959904    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config\") pod \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") "
Sep 05 02:38:47.960022 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.959957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lzzk\" (UniqueName: \"kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk\") pod \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") "
Sep 05 02:38:47.960022 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.959984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls\") pod \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") "
Sep 05 02:38:47.960178 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.960037    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert\") pod \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\" (UID: \"e2cfdf36-52ca-4fd2-be92-8c5e401170a2\") "
Sep 05 02:38:47.960840 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.960538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2cfdf36-52ca-4fd2-be92-8c5e401170a2" (UID: "e2cfdf36-52ca-4fd2-be92-8c5e401170a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:47.960840 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.960593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2cfdf36-52ca-4fd2-be92-8c5e401170a2" (UID: "e2cfdf36-52ca-4fd2-be92-8c5e401170a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:47.961147 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.961116    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2cfdf36-52ca-4fd2-be92-8c5e401170a2" (UID: "e2cfdf36-52ca-4fd2-be92-8c5e401170a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:38:47.962826 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.962803    2582 scope.go:117] "RemoveContainer" containerID="9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"
Sep 05 02:38:47.963284 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:38:47.963256    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8\": container with ID starting with 9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8 not found: ID does not exist" containerID="9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"
Sep 05 02:38:47.963389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.963295    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8"} err="failed to get container status \"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8\": rpc error: code = NotFound desc = could not find container \"9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8\": container with ID starting with 9be04ceaf72bf45d0f996cd93879d40f7514151326f82e8ced9c15a7b13f36a8 not found: ID does not exist"
Sep 05 02:38:47.963389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.963320    2582 scope.go:117] "RemoveContainer" containerID="cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"
Sep 05 02:38:47.963570 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.963549    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk" (OuterVolumeSpecName: "kube-api-access-8lzzk") pod "e2cfdf36-52ca-4fd2-be92-8c5e401170a2" (UID: "e2cfdf36-52ca-4fd2-be92-8c5e401170a2"). InnerVolumeSpecName "kube-api-access-8lzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:38:47.963719 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:38:47.963688    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5\": container with ID starting with cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5 not found: ID does not exist" containerID="cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"
Sep 05 02:38:47.963817 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.963732    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5"} err="failed to get container status \"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5\": rpc error: code = NotFound desc = could not find container \"cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5\": container with ID starting with cae1f5d3f3efaa06793e373673edd2937f9a5ba0c88363c1078ae7c2f95e41c5 not found: ID does not exist"
Sep 05 02:38:47.963885 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.963869    2582 scope.go:117] "RemoveContainer" containerID="1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"
Sep 05 02:38:47.964261 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:38:47.964224    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b\": container with ID starting with 1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b not found: ID does not exist" containerID="1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"
Sep 05 02:38:47.964356 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.964256    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2cfdf36-52ca-4fd2-be92-8c5e401170a2" (UID: "e2cfdf36-52ca-4fd2-be92-8c5e401170a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:38:47.964356 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:47.964258    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b"} err="failed to get container status \"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b\": rpc error: code = NotFound desc = could not find container \"1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b\": container with ID starting with 1a1e8f8c681dde9ad03b74453e0fdcb6a7ded552edc709ed7f62bda96ad3691b not found: ID does not exist"
Sep 05 02:38:48.061359 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.061326    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzzk\" (UniqueName: \"kubernetes.io/projected/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-api-access-8lzzk\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.061537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.061366    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.061537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.061386    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.061537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.061399    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.061537 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.061412    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2cfdf36-52ca-4fd2-be92-8c5e401170a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:38:48.242823 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.242793    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:38:48.249495 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.249475    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4"]
Sep 05 02:38:48.933787 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.933676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerStarted","Data":"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"}
Sep 05 02:38:48.933787 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.933743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerStarted","Data":"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"}
Sep 05 02:38:48.934310 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.934040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:48.934310 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.934183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:48.935307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.935286    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:38:48.953610 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:48.953553    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podStartSLOduration=2.194702577 podStartE2EDuration="5.953536452s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.758833875s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:48.9523517 +0000 UTC m=+1263.274124231" watchObservedRunningTime="2026-09-05 02:38:48.953536452 +0000 UTC m=+1263.275308971"
Sep 05 02:38:49.938685 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:49.938650    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:38:50.183279 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:50.183248    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" path="/var/lib/kubelet/pods/e2cfdf36-52ca-4fd2-be92-8c5e401170a2/volumes"
Sep 05 02:38:51.291356 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:51.291290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:54.943954 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:54.943926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:38:54.944470 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:38:54.944445    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:04.944815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:04.944750    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:14.944815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:14.944776    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:24.945019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:24.944979    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:34.944882 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:34.944837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:44.945019 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:44.944979    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:39:54.944508 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:39:54.944461    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:40:04.945531 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:04.945501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:40:05.671776 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:05.671727    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:40:05.672008 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:05.671977    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container" containerID="cri-o://7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c" gracePeriod=30
Sep 05 02:40:05.672143 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:05.672028    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kube-rbac-proxy" containerID="cri-o://d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb" gracePeriod=30
Sep 05 02:40:06.213520 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:06.213484    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerID="d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb" exitCode=2
Sep 05 02:40:06.213944 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:06.213540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerDied","Data":"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"}
Sep 05 02:40:09.939735 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:09.939630    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 05 02:40:10.511866 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.511840    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:40:10.677475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert\") pod \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") "
Sep 05 02:40:10.677675 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch9cp\" (UniqueName: \"kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp\") pod \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") "
Sep 05 02:40:10.677675 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls\") pod \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") "
Sep 05 02:40:10.677675 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677572    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location\") pod \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") "
Sep 05 02:40:10.677675 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config\") pod \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\" (UID: \"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6\") "
Sep 05 02:40:10.677925 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677902    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" (UID: "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:10.677993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.677965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" (UID: "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:10.678041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.678005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" (UID: "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:10.679977 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.679955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" (UID: "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:10.680072 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.679969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp" (OuterVolumeSpecName: "kube-api-access-ch9cp") pod "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" (UID: "4b5ded96-f97e-4b49-8c32-77b7b1c20aa6"). InnerVolumeSpecName "kube-api-access-ch9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:10.778494 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.778457    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:40:10.778494 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.778492    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9cp\" (UniqueName: \"kubernetes.io/projected/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-api-access-ch9cp\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:40:10.778711 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.778506    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:40:10.778711 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.778519    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:40:10.778711 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:10.778530    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:40:11.232575 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.232543    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerID="7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c" exitCode=0
Sep 05 02:40:11.233025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.232602    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"
Sep 05 02:40:11.233025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.232609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerDied","Data":"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"}
Sep 05 02:40:11.233025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.232661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2" event={"ID":"4b5ded96-f97e-4b49-8c32-77b7b1c20aa6","Type":"ContainerDied","Data":"14fe04ab17980a3938ef3d830ae3a9f14c7011137c923e944c0081bf0a1d60ff"}
Sep 05 02:40:11.233025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.232685    2582 scope.go:117] "RemoveContainer" containerID="d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"
Sep 05 02:40:11.240888 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.240869    2582 scope.go:117] "RemoveContainer" containerID="7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"
Sep 05 02:40:11.247851 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.247806    2582 scope.go:117] "RemoveContainer" containerID="1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"
Sep 05 02:40:11.253788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.253744    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:40:11.254928 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.254913    2582 scope.go:117] "RemoveContainer" containerID="d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"
Sep 05 02:40:11.255178 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:40:11.255160    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb\": container with ID starting with d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb not found: ID does not exist" containerID="d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"
Sep 05 02:40:11.255238 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.255186    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb"} err="failed to get container status \"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb\": rpc error: code = NotFound desc = could not find container \"d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb\": container with ID starting with d41dd8a3ceffec85ab41ee7c72bc9a98939c77519db391ac1fae941bc4f65feb not found: ID does not exist"
Sep 05 02:40:11.255238 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.255210    2582 scope.go:117] "RemoveContainer" containerID="7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"
Sep 05 02:40:11.255444 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:40:11.255428    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c\": container with ID starting with 7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c not found: ID does not exist" containerID="7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"
Sep 05 02:40:11.255507 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.255454    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c"} err="failed to get container status \"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c\": rpc error: code = NotFound desc = could not find container \"7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c\": container with ID starting with 7446db428f246880292201bb44bc51f2e14308ee051f3f9883e7d4ef0e2c519c not found: ID does not exist"
Sep 05 02:40:11.255507 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.255475    2582 scope.go:117] "RemoveContainer" containerID="1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"
Sep 05 02:40:11.255710 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:40:11.255696    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052\": container with ID starting with 1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052 not found: ID does not exist" containerID="1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"
Sep 05 02:40:11.255752 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.255714    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052"} err="failed to get container status \"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052\": rpc error: code = NotFound desc = could not find container \"1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052\": container with ID starting with 1c604fa02f6e185b2615d7e1a0e043f043d031cd82665d9ccd62115d54d9e052 not found: ID does not exist"
Sep 05 02:40:11.259066 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:11.259047    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2"]
Sep 05 02:40:12.184154 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:12.183684    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" path="/var/lib/kubelet/pods/4b5ded96-f97e-4b49-8c32-77b7b1c20aa6/volumes"
Sep 05 02:40:13.179216 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:13.179173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:13.272429 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:13.272396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:18.227869 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.227833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:18.297049 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297013    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:40:18.297263 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297251    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297264    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297278    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="storage-initializer"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297285    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="storage-initializer"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297294    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="storage-initializer"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297300    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="storage-initializer"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297306    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container"
Sep 05 02:40:18.297312 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297311    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297323    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297335    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297340    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297376    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kserve-container"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297383    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5ded96-f97e-4b49-8c32-77b7b1c20aa6" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297388    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kube-rbac-proxy"
Sep 05 02:40:18.297553 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.297394    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cfdf36-52ca-4fd2-be92-8c5e401170a2" containerName="kserve-container"
Sep 05 02:40:18.300345 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.300328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.302798 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.302746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:40:18.302798 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.302757    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:40:18.302972 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.302814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 05 02:40:18.302972 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.302821    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:40:18.303921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.303902    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:40:18.303921 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.303909    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:40:18.304087 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.303935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 05 02:40:18.305592 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.305572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:18.312738 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.312717    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:40:18.371720 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.371681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:18.432658 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.432625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.432868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.432672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.432868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.432748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zh2d\" (UniqueName: \"kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.432868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.432800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.432868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.432826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.533899 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.533813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.533899 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.533858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zh2d\" (UniqueName: \"kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.533899 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.533876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.534194 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.533905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.534194 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:40:18.533978    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 05 02:40:18.534194 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.534046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.534194 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:40:18.534064    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls podName:9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8 nodeName:}" failed. No retries permitted until 2026-09-05 02:40:19.034043382 +0000 UTC m=+1353.355815879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls") pod "isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 05 02:40:18.534412 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.534289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.534557 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.534536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.534666 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.534648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.542830 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.542806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zh2d\" (UniqueName: \"kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:18.635654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:18.635621    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:19.038320 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:19.038279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:19.040935 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:19.040910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-wvx2q\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:19.211100 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:19.211054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:19.484841 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:19.484704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:40:19.487334 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:40:19.487310    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed2ccfc_82b4_4dfa_a31a_26c2595b54b8.slice/crio-1d846a1cb942d0edb8bdc076f1b24a4bc358ec17be530b821999cb16d5b795cc WatchSource:0}: Error finding container 1d846a1cb942d0edb8bdc076f1b24a4bc358ec17be530b821999cb16d5b795cc: Status 404 returned error can't find the container with id 1d846a1cb942d0edb8bdc076f1b24a4bc358ec17be530b821999cb16d5b795cc
Sep 05 02:40:20.267619 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:20.267582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerStarted","Data":"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"}
Sep 05 02:40:20.267824 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:20.267652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerStarted","Data":"1d846a1cb942d0edb8bdc076f1b24a4bc358ec17be530b821999cb16d5b795cc"}
Sep 05 02:40:24.283189 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:24.283145    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerID="669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f" exitCode=0
Sep 05 02:40:24.283686 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:24.283233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerDied","Data":"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"}
Sep 05 02:40:25.289405 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.289374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerStarted","Data":"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"}
Sep 05 02:40:25.289820 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.289434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerStarted","Data":"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"}
Sep 05 02:40:25.289820 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.289753    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:25.289919 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.289901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:25.291155 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.291130    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:40:25.310127 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:25.310065    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podStartSLOduration=3.54353951 podStartE2EDuration="7.310044533s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766505023s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:25.309874864 +0000 UTC m=+1359.631647403" watchObservedRunningTime="2026-09-05 02:40:25.310044533 +0000 UTC m=+1359.631817052"
Sep 05 02:40:26.292718 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:26.292679    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:40:27.297201 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:27.297166    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:40:32.301971 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:32.301938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:40:32.302447 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:32.302419    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:40:42.302426 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:42.302385    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:40:52.302644 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:40:52.302596    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:02.303023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:02.302982    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:12.303053 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:12.303011    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:22.305626 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:22.305587    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:32.302507 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:32.302462    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:41.682353 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:41.682273    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:41:41.682887 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:41.682505    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container" containerID="cri-o://66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422" gracePeriod=30
Sep 05 02:41:41.682887 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:41.682630    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kube-rbac-proxy" containerID="cri-o://69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55" gracePeriod=30
Sep 05 02:41:41.683496 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:41.683473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:41:42.298339 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:42.298292    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 05 02:41:42.561851 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:42.561748    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerID="69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55" exitCode=2
Sep 05 02:41:42.561851 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:42.561808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerDied","Data":"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"}
Sep 05 02:41:46.516152 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.516131    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:41:46.578391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.578298    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerID="66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422" exitCode=0
Sep 05 02:41:46.578391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.578354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"
Sep 05 02:41:46.578391 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.578378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerDied","Data":"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"}
Sep 05 02:41:46.578622 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.578439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q" event={"ID":"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8","Type":"ContainerDied","Data":"1d846a1cb942d0edb8bdc076f1b24a4bc358ec17be530b821999cb16d5b795cc"}
Sep 05 02:41:46.578622 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.578466    2582 scope.go:117] "RemoveContainer" containerID="69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"
Sep 05 02:41:46.586338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.586316    2582 scope.go:117] "RemoveContainer" containerID="66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"
Sep 05 02:41:46.593274 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.593259    2582 scope.go:117] "RemoveContainer" containerID="669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"
Sep 05 02:41:46.600024 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.600009    2582 scope.go:117] "RemoveContainer" containerID="69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"
Sep 05 02:41:46.600310 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:41:46.600294    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55\": container with ID starting with 69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55 not found: ID does not exist" containerID="69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"
Sep 05 02:41:46.600389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.600323    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55"} err="failed to get container status \"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55\": rpc error: code = NotFound desc = could not find container \"69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55\": container with ID starting with 69b3674b3fd8796257a18b212b60c4470230f970ec9a25c8e99e1d513cd74b55 not found: ID does not exist"
Sep 05 02:41:46.600389 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.600347    2582 scope.go:117] "RemoveContainer" containerID="66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"
Sep 05 02:41:46.600618 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:41:46.600601    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422\": container with ID starting with 66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422 not found: ID does not exist" containerID="66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"
Sep 05 02:41:46.600671 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.600623    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422"} err="failed to get container status \"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422\": rpc error: code = NotFound desc = could not find container \"66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422\": container with ID starting with 66eecda34f0e468354ea412f8034aefe2cf16bd3ebe7cabd45cb19e6e0e5b422 not found: ID does not exist"
Sep 05 02:41:46.600671 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.600638    2582 scope.go:117] "RemoveContainer" containerID="669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"
Sep 05 02:41:46.601031 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:41:46.601014    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f\": container with ID starting with 669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f not found: ID does not exist" containerID="669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"
Sep 05 02:41:46.601083 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.601037    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f"} err="failed to get container status \"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f\": rpc error: code = NotFound desc = could not find container \"669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f\": container with ID starting with 669fcebbe7692459a92ba5364e13c644855e16ecb43330724bee0a2d1d5abc4f not found: ID does not exist"
Sep 05 02:41:46.684445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") pod \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") "
Sep 05 02:41:46.684445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location\") pod \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") "
Sep 05 02:41:46.684669 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config\") pod \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") "
Sep 05 02:41:46.684669 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zh2d\" (UniqueName: \"kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d\") pod \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") "
Sep 05 02:41:46.684669 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert\") pod \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\" (UID: \"9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8\") "
Sep 05 02:41:46.684865 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:41:46.684957 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.684933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:46.685046 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.685025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:46.687285 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.687266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d" (OuterVolumeSpecName: "kube-api-access-9zh2d") pod "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8"). InnerVolumeSpecName "kube-api-access-9zh2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:46.687341 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.687276    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" (UID: "9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:46.785746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.785700    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zh2d\" (UniqueName: \"kubernetes.io/projected/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-api-access-9zh2d\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:41:46.785746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.785742    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:41:46.785746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.785754    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:41:46.785746 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.785786    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:41:46.786023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.785795    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:41:46.899137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.899104    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:41:46.902868 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:46.902845    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q"]
Sep 05 02:41:48.182575 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.182538    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" path="/var/lib/kubelet/pods/9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8/volumes"
Sep 05 02:41:48.365980 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.365945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:48.455376 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455289    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:41:48.455634 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455618    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="storage-initializer"
Sep 05 02:41:48.455717 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455637    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="storage-initializer"
Sep 05 02:41:48.455717 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455673    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kube-rbac-proxy"
Sep 05 02:41:48.455717 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455683    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kube-rbac-proxy"
Sep 05 02:41:48.455717 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container"
Sep 05 02:41:48.455717 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455701    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container"
Sep 05 02:41:48.456008 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455787    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kube-rbac-proxy"
Sep 05 02:41:48.456008 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.455800    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed2ccfc-82b4-4dfa-a31a-26c2595b54b8" containerName="kserve-container"
Sep 05 02:41:48.460034 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.460013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.462661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.462641    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:41:48.462661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.462667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:41:48.462870 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.462719    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:41:48.462870 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.462740    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:41:48.462870 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.462741    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 05 02:41:48.463689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.463675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:41:48.463689 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.463682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 05 02:41:48.468438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.468419    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:41:48.601218 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.601187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.601218 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.601221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8667g\" (UniqueName: \"kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.601425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.601247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.601425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.601354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.601425 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.601392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.702509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.702476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8667g\" (UniqueName: \"kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.702509 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.702513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.702755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.702552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.702755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.702576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.702755 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.702722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.703103 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.703083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.703188 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.703156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.703282 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.703262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.705269 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.705248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.710933 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.710883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8667g\" (UniqueName: \"kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-m9vwm\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:48.770947 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:48.770898    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:49.045735 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:49.045659    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:41:49.051484 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:41:49.049538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f726fe6_8fd3_40df_9472_f6bd0353a522.slice/crio-0f4789fa8a79a5b92e134f2d966523d7240d722690b08d075d41d57ed95866e8 WatchSource:0}: Error finding container 0f4789fa8a79a5b92e134f2d966523d7240d722690b08d075d41d57ed95866e8: Status 404 returned error can't find the container with id 0f4789fa8a79a5b92e134f2d966523d7240d722690b08d075d41d57ed95866e8
Sep 05 02:41:49.591673 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:49.591635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerStarted","Data":"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"}
Sep 05 02:41:49.592094 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:49.591712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerStarted","Data":"0f4789fa8a79a5b92e134f2d966523d7240d722690b08d075d41d57ed95866e8"}
Sep 05 02:41:53.607191 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:53.607150    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerID="c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a" exitCode=0
Sep 05 02:41:53.607564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:53.607219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerDied","Data":"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"}
Sep 05 02:41:54.613187 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:54.613156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerStarted","Data":"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"}
Sep 05 02:41:54.613583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:54.613213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerStarted","Data":"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"}
Sep 05 02:41:54.613583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:54.613394    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:54.632355 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:54.632307    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podStartSLOduration=2.956535096 podStartE2EDuration="6.632290547s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675755451s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:54.630432843 +0000 UTC m=+1448.952205361" watchObservedRunningTime="2026-09-05 02:41:54.632290547 +0000 UTC m=+1448.954063066"
Sep 05 02:41:55.617416 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:55.617385    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:41:55.618664 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:55.618632    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:41:56.620532 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:41:56.620491    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:01.626069 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:01.626039    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:42:01.626543 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:01.626518    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:11.626684 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:11.626634    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:21.627262 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:21.627219    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:31.626922 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:31.626873    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:41.626744 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:41.626703    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:46.211607 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:46.211576    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:42:46.213860 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:46.213840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:42:51.627023 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:42:51.626977    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:43:01.627222 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:01.627176    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:43:11.575621 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:11.575546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:11.627269 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:11.627245    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:43:18.588322 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.588286    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:43:18.588825 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.588527    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" containerID="cri-o://caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997" gracePeriod=30
Sep 05 02:43:18.588825 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.588557    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kube-rbac-proxy" containerID="cri-o://a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339" gracePeriod=30
Sep 05 02:43:18.617714 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.617684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:18.915007 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.914978    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerID="a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339" exitCode=2
Sep 05 02:43:18.915180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:18.915045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerDied","Data":"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"}
Sep 05 02:43:21.620702 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:21.620662    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 05 02:43:21.627009 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:21.626983    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:43:23.430528 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.430505    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:43:23.606040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config\") pod \"4f726fe6-8fd3-40df-9472-f6bd0353a522\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") "
Sep 05 02:43:23.606229 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606066    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert\") pod \"4f726fe6-8fd3-40df-9472-f6bd0353a522\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") "
Sep 05 02:43:23.606229 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606097    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8667g\" (UniqueName: \"kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g\") pod \"4f726fe6-8fd3-40df-9472-f6bd0353a522\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") "
Sep 05 02:43:23.606229 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location\") pod \"4f726fe6-8fd3-40df-9472-f6bd0353a522\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") "
Sep 05 02:43:23.606229 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls\") pod \"4f726fe6-8fd3-40df-9472-f6bd0353a522\" (UID: \"4f726fe6-8fd3-40df-9472-f6bd0353a522\") "
Sep 05 02:43:23.606460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f726fe6-8fd3-40df-9472-f6bd0353a522" (UID: "4f726fe6-8fd3-40df-9472-f6bd0353a522"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:43:23.606460 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f726fe6-8fd3-40df-9472-f6bd0353a522" (UID: "4f726fe6-8fd3-40df-9472-f6bd0353a522"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:23.606564 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.606519    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f726fe6-8fd3-40df-9472-f6bd0353a522" (UID: "4f726fe6-8fd3-40df-9472-f6bd0353a522"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:43:23.608390 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.608369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f726fe6-8fd3-40df-9472-f6bd0353a522" (UID: "4f726fe6-8fd3-40df-9472-f6bd0353a522"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:43:23.608489 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.608470    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g" (OuterVolumeSpecName: "kube-api-access-8667g") pod "4f726fe6-8fd3-40df-9472-f6bd0353a522" (UID: "4f726fe6-8fd3-40df-9472-f6bd0353a522"). InnerVolumeSpecName "kube-api-access-8667g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:43:23.707521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.707477    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8667g\" (UniqueName: \"kubernetes.io/projected/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-api-access-8667g\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:43:23.707521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.707513    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f726fe6-8fd3-40df-9472-f6bd0353a522-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:43:23.707521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.707529    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f726fe6-8fd3-40df-9472-f6bd0353a522-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:43:23.707789 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.707541    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:43:23.707789 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.707553    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f726fe6-8fd3-40df-9472-f6bd0353a522-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:43:23.934464 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.934371    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerID="caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997" exitCode=0
Sep 05 02:43:23.934464 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.934419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerDied","Data":"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"}
Sep 05 02:43:23.934464 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.934432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"
Sep 05 02:43:23.934705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.934474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm" event={"ID":"4f726fe6-8fd3-40df-9472-f6bd0353a522","Type":"ContainerDied","Data":"0f4789fa8a79a5b92e134f2d966523d7240d722690b08d075d41d57ed95866e8"}
Sep 05 02:43:23.934705 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.934501    2582 scope.go:117] "RemoveContainer" containerID="a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"
Sep 05 02:43:23.947158 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.947139    2582 scope.go:117] "RemoveContainer" containerID="caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"
Sep 05 02:43:23.954305 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.954287    2582 scope.go:117] "RemoveContainer" containerID="c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"
Sep 05 02:43:23.960035 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.960015    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:43:23.963397 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.963375    2582 scope.go:117] "RemoveContainer" containerID="a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"
Sep 05 02:43:23.963746 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:43:23.963727    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339\": container with ID starting with a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339 not found: ID does not exist" containerID="a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"
Sep 05 02:43:23.963819 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.963756    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339"} err="failed to get container status \"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339\": rpc error: code = NotFound desc = could not find container \"a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339\": container with ID starting with a594f4841b436cf6763a9976c50fe8553ef154dad07ddb760188ebd7f4716339 not found: ID does not exist"
Sep 05 02:43:23.963819 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.963805    2582 scope.go:117] "RemoveContainer" containerID="caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"
Sep 05 02:43:23.964074 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:43:23.964056    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997\": container with ID starting with caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997 not found: ID does not exist" containerID="caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"
Sep 05 02:43:23.964128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.964082    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997"} err="failed to get container status \"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997\": rpc error: code = NotFound desc = could not find container \"caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997\": container with ID starting with caeacb53b9af4c845ff9505a8ddf23ea6eb64e6ab7159565c693f7b718de6997 not found: ID does not exist"
Sep 05 02:43:23.964128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.964096    2582 scope.go:117] "RemoveContainer" containerID="c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"
Sep 05 02:43:23.964198 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.964180    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm"]
Sep 05 02:43:23.964369 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:43:23.964353    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a\": container with ID starting with c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a not found: ID does not exist" containerID="c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"
Sep 05 02:43:23.964406 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:23.964376    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a"} err="failed to get container status \"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a\": rpc error: code = NotFound desc = could not find container \"c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a\": container with ID starting with c045ddb67207410041091bb4d96b0a4868717d5e5be0ee930b768a921150ce8a not found: ID does not exist"
Sep 05 02:43:24.184099 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:24.184063    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" path="/var/lib/kubelet/pods/4f726fe6-8fd3-40df-9472-f6bd0353a522/volumes"
Sep 05 02:43:32.644976 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:32.644927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:35.117187 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:35.117153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:42.715570 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715528    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715908    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715926    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715941    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="storage-initializer"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715949    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="storage-initializer"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715962    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kube-rbac-proxy"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.715969    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kube-rbac-proxy"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.716046    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kserve-container"
Sep 05 02:43:42.716070 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.716058    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f726fe6-8fd3-40df-9472-f6bd0353a522" containerName="kube-rbac-proxy"
Sep 05 02:43:42.720856 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.720834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.723329 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723308    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:43:42.723445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:43:42.723445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723344    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:43:42.723445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:43:42.723591 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 05 02:43:42.723661 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.723647    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 05 02:43:42.724079 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.724064    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:43:42.732997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.732976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:43:42.736913 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.736893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.737011 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.736932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.737011 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.736961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.737088 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.737014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.737088 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.737030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdfz\" (UniqueName: \"kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.837815 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.837782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.837988 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.837843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.837988 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.837889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.837988 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.837931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.837988 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.837956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdfz\" (UniqueName: \"kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.838218 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:43:42.838062    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 05 02:43:42.838218 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:43:42.838140    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls podName:671a2933-dfe9-4df9-bda8-f4ba270b3c5f nodeName:}" failed. No retries permitted until 2026-09-05 02:43:43.338119089 +0000 UTC m=+1557.659891586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls") pod "isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 05 02:43:42.838218 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.838190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.838583 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.838562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.838654 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.838565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:42.847122 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:42.847102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdfz\" (UniqueName: \"kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:43.341330 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.341289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:43.344030 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.344004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-84b648dc8b-hj622\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:43.349781 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.349730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:43.463244 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.463213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:43.630967 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.630935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:43.913428 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:43.913340    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:43:43.916213 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:43:43.916189    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671a2933_dfe9_4df9_bda8_f4ba270b3c5f.slice/crio-f2134876c067e7981f844cb18e54a4dfb29315329a6478b7bd963845f1bc94d3 WatchSource:0}: Error finding container f2134876c067e7981f844cb18e54a4dfb29315329a6478b7bd963845f1bc94d3: Status 404 returned error can't find the container with id f2134876c067e7981f844cb18e54a4dfb29315329a6478b7bd963845f1bc94d3
Sep 05 02:43:44.012818 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:44.012781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerStarted","Data":"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"}
Sep 05 02:43:44.012989 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:44.012847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerStarted","Data":"f2134876c067e7981f844cb18e54a4dfb29315329a6478b7bd963845f1bc94d3"}
Sep 05 02:43:48.027579 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:48.027493    2582 generic.go:356] "Generic (PLEG): container finished" podID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerID="16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa" exitCode=0
Sep 05 02:43:48.027579 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:48.027570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerDied","Data":"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"}
Sep 05 02:43:49.033907 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.033866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerStarted","Data":"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"}
Sep 05 02:43:49.034285 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.033942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerStarted","Data":"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"}
Sep 05 02:43:49.034285 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.033970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerStarted","Data":"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"}
Sep 05 02:43:49.034285 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.034278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:49.034384 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.034328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:49.035711 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.035683    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:43:49.066245 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:49.066190    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podStartSLOduration=3.281055445 podStartE2EDuration="7.066173381s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785117936s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:49.064517453 +0000 UTC m=+1563.386289992" watchObservedRunningTime="2026-09-05 02:43:49.066173381 +0000 UTC m=+1563.387945899"
Sep 05 02:43:50.037556 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:50.037519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:50.038045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:50.037702    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:43:50.038906 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:50.038883    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:43:51.041469 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:51.041426    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:43:51.041896 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:51.041820    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:43:56.046013 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:56.045977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:43:56.046487 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:56.046461    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:43:56.046837 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:43:56.046812    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:06.046778 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:06.046714    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:06.047196 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:06.047090    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:16.046959 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:16.046911    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:16.047416 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:16.047394    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:26.047446 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:26.047395    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:26.047913 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:26.047854    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:36.046478 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:36.046435    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:36.046916 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:36.046821    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:46.047213 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:46.047113    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:46.047631 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:46.047606    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:56.047275 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:56.047230    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:56.047751 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:56.047662    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:44:57.178378 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:57.178333    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:44:57.178880 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:44:57.178836    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:45:07.179547 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:07.179516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:07.179957 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:07.179817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:17.086727 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:17.086698    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:18.000324 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.000289    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:45:18.000657 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.000606    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" containerID="cri-o://ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b" gracePeriod=30
Sep 05 02:45:18.000801 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.000652    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" containerID="cri-o://8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86" gracePeriod=30
Sep 05 02:45:18.000897 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.000666    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" containerID="cri-o://5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390" gracePeriod=30
Sep 05 02:45:18.047576 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.047543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:18.118180 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.118139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:18.362056 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.362026    2582 generic.go:356] "Generic (PLEG): container finished" podID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerID="5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390" exitCode=2
Sep 05 02:45:18.362227 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:18.362069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerDied","Data":"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"}
Sep 05 02:45:19.132163 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:19.132126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:19.230997 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:19.230959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:21.042086 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:21.042048    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:23.380380 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:23.380347    2582 generic.go:356] "Generic (PLEG): container finished" podID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerID="ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b" exitCode=0
Sep 05 02:45:23.380757 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:23.380418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerDied","Data":"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"}
Sep 05 02:45:26.042017 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:26.041976    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:27.179110 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:27.179066    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:45:27.179635 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:27.179386    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:45:31.042349 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:31.042306    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:31.042742 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:31.042462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:35.057454 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.057427    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:35.084402 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.084377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:35.097662 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.097638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:35.109369 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.109352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:35.935457 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.935430    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:35.947597 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.947561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:35.957623 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.957599    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:35.968806 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:35.968788    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:36.041778 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:36.041720    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:36.795296 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:36.795268    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:36.817267 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:36.817241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:36.871215 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:36.871185    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:36.913126 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:36.913105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:37.178663 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.178620    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:45:37.178993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.178972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:45:37.723880 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.723838    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:37.735657 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.735633    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:37.745978 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.745956    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:37.756306 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:37.756285    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:38.568279 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:38.568246    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:38.580660 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:38.580636    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:38.593979 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:38.593955    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:38.605674 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:38.605650    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:39.438559 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:39.438532    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:39.451273 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:39.451251    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:39.464681 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:39.464655    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:39.479560 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:39.479539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:40.295127 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:40.295101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:40.305742 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:40.305718    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:40.331621 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:40.331596    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:40.369291 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:40.369266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:41.042435 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:41.042397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:41.212483 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:41.212451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:41.224757 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:41.224736    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:41.240510 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:41.240484    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:41.251307 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:41.251287    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:42.061442 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.061414    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:42.074465 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.074444    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:42.085344 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.085323    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:42.095788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.095755    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:42.895040 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.895016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:42.906952 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.906931    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:42.918013 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.917996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:42.932041 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:42.932011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:43.735143 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:43.735108    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:43.747076 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:43.747052    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:43.757030 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:43.757009    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:43.767338 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:43.767299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:44.567495 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:44.567468    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:44.578447 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:44.578415    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:44.588217 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:44.588199    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:44.598847 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:44.598829    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:45.406514 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:45.406488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:45.419166 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:45.419142    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:45.430585 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:45.430561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:45.441802 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:45.441779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:46.042433 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:46.042390    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 05 02:45:46.342980 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:46.342960    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kserve-container/0.log"
Sep 05 02:45:46.375331 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:46.375303    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/kube-rbac-proxy/0.log"
Sep 05 02:45:46.425451 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:46.425426    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/agent/0.log"
Sep 05 02:45:46.442106 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:46.442074    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-84b648dc8b-hj622_671a2933-dfe9-4df9-bda8-f4ba270b3c5f/storage-initializer/0.log"
Sep 05 02:45:47.178604 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:47.178545    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:45:47.178866 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:47.178833    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:45:47.179128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:47.179111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:47.179239 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:47.179227    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:48.147253 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.147229    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:48.240792 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.240748    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config\") pod \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") "
Sep 05 02:45:48.240792 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.240799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") pod \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") "
Sep 05 02:45:48.241043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.240821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssdfz\" (UniqueName: \"kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz\") pod \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") "
Sep 05 02:45:48.241043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.240856    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location\") pod \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") "
Sep 05 02:45:48.241043 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.240883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert\") pod \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\" (UID: \"671a2933-dfe9-4df9-bda8-f4ba270b3c5f\") "
Sep 05 02:45:48.241224 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.241196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "671a2933-dfe9-4df9-bda8-f4ba270b3c5f" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:48.241224 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.241203    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "671a2933-dfe9-4df9-bda8-f4ba270b3c5f" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:48.241344 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.241316    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "671a2933-dfe9-4df9-bda8-f4ba270b3c5f" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:48.243109 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.243079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "671a2933-dfe9-4df9-bda8-f4ba270b3c5f" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:48.243240 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.243220    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz" (OuterVolumeSpecName: "kube-api-access-ssdfz") pod "671a2933-dfe9-4df9-bda8-f4ba270b3c5f" (UID: "671a2933-dfe9-4df9-bda8-f4ba270b3c5f"). InnerVolumeSpecName "kube-api-access-ssdfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:48.342417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.342325    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kserve-provision-location\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:45:48.342417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.342354    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-cabundle-cert\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:45:48.342417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.342366    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:45:48.342417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.342375    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-proxy-tls\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:45:48.342417 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.342383    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdfz\" (UniqueName: \"kubernetes.io/projected/671a2933-dfe9-4df9-bda8-f4ba270b3c5f-kube-api-access-ssdfz\") on node \"ip-10-0-131-142.ec2.internal\" DevicePath \"\""
Sep 05 02:45:48.470351 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.470313    2582 generic.go:356] "Generic (PLEG): container finished" podID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerID="8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86" exitCode=0
Sep 05 02:45:48.470351 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.470352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerDied","Data":"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"}
Sep 05 02:45:48.470568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.470382    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"
Sep 05 02:45:48.470568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.470408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622" event={"ID":"671a2933-dfe9-4df9-bda8-f4ba270b3c5f","Type":"ContainerDied","Data":"f2134876c067e7981f844cb18e54a4dfb29315329a6478b7bd963845f1bc94d3"}
Sep 05 02:45:48.470568 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.470451    2582 scope.go:117] "RemoveContainer" containerID="8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"
Sep 05 02:45:48.478675 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.478656    2582 scope.go:117] "RemoveContainer" containerID="5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"
Sep 05 02:45:48.485737 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.485720    2582 scope.go:117] "RemoveContainer" containerID="ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"
Sep 05 02:45:48.493522 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.493193    2582 scope.go:117] "RemoveContainer" containerID="16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"
Sep 05 02:45:48.494000 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.493978    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:45:48.498269 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.498249    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622"]
Sep 05 02:45:48.500754 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.500736    2582 scope.go:117] "RemoveContainer" containerID="8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"
Sep 05 02:45:48.501046 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:45:48.501028    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86\": container with ID starting with 8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86 not found: ID does not exist" containerID="8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"
Sep 05 02:45:48.501107 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501054    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86"} err="failed to get container status \"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86\": rpc error: code = NotFound desc = could not find container \"8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86\": container with ID starting with 8a3a826348420d04268fc1018178f8383f2393b7f92088a8ddabcc2b2323fd86 not found: ID does not exist"
Sep 05 02:45:48.501107 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501071    2582 scope.go:117] "RemoveContainer" containerID="5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"
Sep 05 02:45:48.501330 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:45:48.501316    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390\": container with ID starting with 5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390 not found: ID does not exist" containerID="5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"
Sep 05 02:45:48.501385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501334    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390"} err="failed to get container status \"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390\": rpc error: code = NotFound desc = could not find container \"5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390\": container with ID starting with 5b45e1ea3829b5971e86ba10ebb901993bb8171680b4c20e76ceb4c5b1f99390 not found: ID does not exist"
Sep 05 02:45:48.501385 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501346    2582 scope.go:117] "RemoveContainer" containerID="ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"
Sep 05 02:45:48.501550 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:45:48.501534    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b\": container with ID starting with ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b not found: ID does not exist" containerID="ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"
Sep 05 02:45:48.501608 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501555    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b"} err="failed to get container status \"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b\": rpc error: code = NotFound desc = could not find container \"ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b\": container with ID starting with ff1ae0e7433d362457c024a3f16bca96d11553d3968ce0c2e9fc8d6de3f17b7b not found: ID does not exist"
Sep 05 02:45:48.501608 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501569    2582 scope.go:117] "RemoveContainer" containerID="16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"
Sep 05 02:45:48.501838 ip-10-0-131-142 kubenswrapper[2582]: E0905 02:45:48.501816    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa\": container with ID starting with 16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa not found: ID does not exist" containerID="16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"
Sep 05 02:45:48.501903 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:48.501835    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa"} err="failed to get container status \"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa\": rpc error: code = NotFound desc = could not find container \"16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa\": container with ID starting with 16b80bb87577be6c742718930597dfcfd0e89768cabfd628a60659998b847dfa not found: ID does not exist"
Sep 05 02:45:50.182811 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.182781    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" path="/var/lib/kubelet/pods/671a2933-dfe9-4df9-bda8-f4ba270b3c5f/volumes"
Sep 05 02:45:50.185888 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.185868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335433    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s672k/must-gather-2wnm7"]
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335667    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335677    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335685    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335690    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335709    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="storage-initializer"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335716    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="storage-initializer"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335721    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335726    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335777    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kserve-container"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335787    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="kube-rbac-proxy"
Sep 05 02:45:50.339475 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.335794    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="671a2933-dfe9-4df9-bda8-f4ba270b3c5f" containerName="agent"
Sep 05 02:45:50.340321 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.340305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.342989 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.342969    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s672k\"/\"openshift-service-ca.crt\""
Sep 05 02:45:50.343337 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.343319    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s672k\"/\"default-dockercfg-d5z68\""
Sep 05 02:45:50.343653 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.343639    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s672k\"/\"kube-root-ca.crt\""
Sep 05 02:45:50.351529 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.351510    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/must-gather-2wnm7"]
Sep 05 02:45:50.357483 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.357454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2dv\" (UniqueName: \"kubernetes.io/projected/d228abc2-67c1-4938-be89-5ef2e080435a-kube-api-access-mt2dv\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.357573 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.357493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d228abc2-67c1-4938-be89-5ef2e080435a-must-gather-output\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.360073 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.360050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:50.425020 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.424990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:50.458900 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.458822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2dv\" (UniqueName: \"kubernetes.io/projected/d228abc2-67c1-4938-be89-5ef2e080435a-kube-api-access-mt2dv\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.458900 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.458855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d228abc2-67c1-4938-be89-5ef2e080435a-must-gather-output\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.459140 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.459124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d228abc2-67c1-4938-be89-5ef2e080435a-must-gather-output\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.467877 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.467852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2dv\" (UniqueName: \"kubernetes.io/projected/d228abc2-67c1-4938-be89-5ef2e080435a-kube-api-access-mt2dv\") pod \"must-gather-2wnm7\" (UID: \"d228abc2-67c1-4938-be89-5ef2e080435a\") " pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.649715 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.649677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/must-gather-2wnm7"
Sep 05 02:45:50.919953 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.919922    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/must-gather-2wnm7"]
Sep 05 02:45:50.923098 ip-10-0-131-142 kubenswrapper[2582]: W0905 02:45:50.923068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd228abc2_67c1_4938_be89_5ef2e080435a.slice/crio-c61a22cf68868be1fbe3c89b8572e9621c92300dc9f81b7f29b3d14a5618b83c WatchSource:0}: Error finding container c61a22cf68868be1fbe3c89b8572e9621c92300dc9f81b7f29b3d14a5618b83c: Status 404 returned error can't find the container with id c61a22cf68868be1fbe3c89b8572e9621c92300dc9f81b7f29b3d14a5618b83c
Sep 05 02:45:50.924940 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:50.924920    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:45:51.482714 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:51.482679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/must-gather-2wnm7" event={"ID":"d228abc2-67c1-4938-be89-5ef2e080435a","Type":"ContainerStarted","Data":"c61a22cf68868be1fbe3c89b8572e9621c92300dc9f81b7f29b3d14a5618b83c"}
Sep 05 02:45:52.490473 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:52.490411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/must-gather-2wnm7" event={"ID":"d228abc2-67c1-4938-be89-5ef2e080435a","Type":"ContainerStarted","Data":"bce97ea305d6d01623003b0476a00e31f14ff24281fcec121e4cdd5198da1e05"}
Sep 05 02:45:52.490993 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:52.490496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/must-gather-2wnm7" event={"ID":"d228abc2-67c1-4938-be89-5ef2e080435a","Type":"ContainerStarted","Data":"dca8f6a8e8a7e9b6cb1290d9aa491ab36d01ab586bdb2740868d3dc1ff476735"}
Sep 05 02:45:53.175810 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:53.175780    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8dhc_79bd5ce1-cb17-4035-984a-dfc16c53b637/global-pull-secret-syncer/0.log"
Sep 05 02:45:53.219217 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:53.219159    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log"
Sep 05 02:45:53.295305 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:53.295272    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-142.ec2.internal_58007dd99525496ea22ff71e4a80cd63/haproxy/0.log"
Sep 05 02:45:56.267250 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:56.267182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:56.325894 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:56.325859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cplkx_84933ea4-b102-42f9-b2e9-5f47c655130c/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:57.757278 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:57.757241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm695_203d1a45-4272-45a2-b07e-9d246b1aa368/node-exporter/0.log"
Sep 05 02:45:57.808137 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:57.808055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm695_203d1a45-4272-45a2-b07e-9d246b1aa368/kube-rbac-proxy/0.log"
Sep 05 02:45:57.831843 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:45:57.831814    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm695_203d1a45-4272-45a2-b07e-9d246b1aa368/init-textfile/0.log"
Sep 05 02:46:01.619045 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.618983    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s672k/must-gather-2wnm7" podStartSLOduration=10.876036147 podStartE2EDuration="11.618948551s" totalImagesPullingTime="742.912404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:45:52.508128125 +0000 UTC m=+1686.829900645" watchObservedRunningTime="2026-09-05 02:46:01.618948551 +0000 UTC m=+1695.940721074"
Sep 05 02:46:01.619788 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.619746    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"]
Sep 05 02:46:01.625050 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.625023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.634945 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.634916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"]
Sep 05 02:46:01.655444 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.655411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-lib-modules\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.655634 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.655464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-sys\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.655634 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.655529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-proc\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.655634 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.655591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cf8\" (UniqueName: \"kubernetes.io/projected/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-kube-api-access-z4cf8\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.655852 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.655640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-podres\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-lib-modules\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756438 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-sys\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-proc\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cf8\" (UniqueName: \"kubernetes.io/projected/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-kube-api-access-z4cf8\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-podres\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-sys\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-lib-modules\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-proc\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.756677 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.756638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-podres\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.764882 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.764850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cf8\" (UniqueName: \"kubernetes.io/projected/4c1a5678-fe4c-434e-89e2-f28cc7fe7216-kube-api-access-z4cf8\") pod \"perf-node-gather-daemonset-k2lnf\" (UID: \"4c1a5678-fe4c-434e-89e2-f28cc7fe7216\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:01.936609 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:01.936518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:02.230625 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.230541    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"]
Sep 05 02:46:02.434313 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.434266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7p6tl_33c51d59-1ee4-456c-acdf-6f4471457da0/dns/0.log"
Sep 05 02:46:02.458170 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.458139    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7p6tl_33c51d59-1ee4-456c-acdf-6f4471457da0/kube-rbac-proxy/0.log"
Sep 05 02:46:02.541375 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.541282    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lclfz_0b1f3509-edfb-416a-b956-b469a9841b91/dns-node-resolver/0.log"
Sep 05 02:46:02.544503 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.544481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf" event={"ID":"4c1a5678-fe4c-434e-89e2-f28cc7fe7216","Type":"ContainerStarted","Data":"d89da13ca2d15d857e2b7c1f485145a0c318c38a1844eede0738ac6f83d08f74"}
Sep 05 02:46:02.544637 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.544522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf" event={"ID":"4c1a5678-fe4c-434e-89e2-f28cc7fe7216","Type":"ContainerStarted","Data":"c76409c00e9e93590eef5c915285d598c348378e62bf0e12af1a36ca2872b310"}
Sep 05 02:46:02.544637 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.544617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:02.560025 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:02.559965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf" podStartSLOduration=1.559943172 podStartE2EDuration="1.559943172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:02.559164351 +0000 UTC m=+1696.880936883" watchObservedRunningTime="2026-09-05 02:46:02.559943172 +0000 UTC m=+1696.881715692"
Sep 05 02:46:03.093837 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:03.093810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bh2bf_629c6726-442d-4ee3-99a5-323e477b1e53/node-ca/0.log"
Sep 05 02:46:03.847643 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:03.847619    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c954d494-7nrt6_131bce09-5bd6-4376-9aff-440919804a4d/router/0.log"
Sep 05 02:46:04.226459 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:04.226380    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8tn8_206465c0-c8d6-4238-b7aa-14409ee968f1/serve-healthcheck-canary/0.log"
Sep 05 02:46:04.606416 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:04.606391    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6f75_bc698016-9978-4b50-9bb0-b80f78de1181/kube-rbac-proxy/0.log"
Sep 05 02:46:04.626860 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:04.626837    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6f75_bc698016-9978-4b50-9bb0-b80f78de1181/exporter/0.log"
Sep 05 02:46:04.648384 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:04.648358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6f75_bc698016-9978-4b50-9bb0-b80f78de1181/extractor/0.log"
Sep 05 02:46:07.024730 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:07.024699    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-577b5fb6f8-cnxhl_6007671b-bfff-487e-bdf5-6e8123379edf/manager/0.log"
Sep 05 02:46:07.054439 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:07.054407    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-9v27f_c3d9108f-ba55-4ff7-a565-eed6d0289c0a/manager/0.log"
Sep 05 02:46:08.561867 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:08.561829    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-k2lnf"
Sep 05 02:46:11.850400 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:11.850375    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hw5qt_3a5650fe-be9b-4c6f-8428-8193261874ad/migrator/0.log"
Sep 05 02:46:11.881920 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:11.881893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hw5qt_3a5650fe-be9b-4c6f-8428-8193261874ad/graceful-termination/0.log"
Sep 05 02:46:13.365647 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.365614    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/kube-multus-additional-cni-plugins/0.log"
Sep 05 02:46:13.390521 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.390492    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/egress-router-binary-copy/0.log"
Sep 05 02:46:13.413421 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.413392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/cni-plugins/0.log"
Sep 05 02:46:13.435958 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.435934    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/bond-cni-plugin/0.log"
Sep 05 02:46:13.458853 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.458831    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/routeoverride-cni/0.log"
Sep 05 02:46:13.482200 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.482171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/whereabouts-cni-bincopy/0.log"
Sep 05 02:46:13.505088 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.505057    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpws9_bf8efa68-e2e0-478d-8244-a1433ec4c1a2/whereabouts-cni/0.log"
Sep 05 02:46:13.714590 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.714558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bw72b_d06cc8bd-5304-4d9b-8737-48a1f5370e22/kube-multus/0.log"
Sep 05 02:46:13.789933 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.789902    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2nhpg_37c79c34-5580-45bb-85b2-b82afd3dd6e2/network-metrics-daemon/0.log"
Sep 05 02:46:13.810068 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:13.810045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2nhpg_37c79c34-5580-45bb-85b2-b82afd3dd6e2/kube-rbac-proxy/0.log"
Sep 05 02:46:14.659539 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.659509    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-controller/0.log"
Sep 05 02:46:14.677445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.677415    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/1.log"
Sep 05 02:46:14.677445 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.677444    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovn-acl-logging/0.log"
Sep 05 02:46:14.697656 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.697612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/kube-rbac-proxy-node/0.log"
Sep 05 02:46:14.720585 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.720554    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 02:46:14.739621 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.739594    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/northd/0.log"
Sep 05 02:46:14.762097 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.762071    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/nbdb/0.log"
Sep 05 02:46:14.783003 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.782983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/sbdb/0.log"
Sep 05 02:46:14.824128 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:14.824090    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b72d9_552d8eb0-1a34-4f95-af5f-3c863be9df7f/ovnkube-controller/0.log"
Sep 05 02:46:16.251720 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:16.251695    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xxl5g_45394a99-f247-4de3-aabb-ce4fd06ed5f0/network-check-target-container/0.log"
Sep 05 02:46:17.207695 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:17.207672    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gjfmw_bd978685-2990-4515-8493-ae261615757d/iptables-alerter/0.log"
Sep 05 02:46:17.934206 ip-10-0-131-142 kubenswrapper[2582]: I0905 02:46:17.934178    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kfh4z_18ca5a4f-3782-4a35-9e89-2c34cc0f96b7/tuned/0.log"