Sep 17 17:38:43.937712 ip-10-0-134-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:38:43.937727 ip-10-0-134-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:38:43.937738 ip-10-0-134-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:38:43.937957 ip-10-0-134-156 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:38:53.956152 ip-10-0-134-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:38:53.956169 ip-10-0-134-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3f95f49de6054eae96010d7accd1c8d5 --
Sep 17 17:40:59.695005 ip-10-0-134-156 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:41:03.141521 ip-10-0-134-156 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:03.141521 ip-10-0-134-156 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:41:03.141521 ip-10-0-134-156 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:03.141521 ip-10-0-134-156 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:41:03.141521 ip-10-0-134-156 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:03.142464 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.141586    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:41:03.145454 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145436    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.145454 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145454    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145457    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145461    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145465    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145468    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145471    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145474    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145477    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145479    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145482    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145485    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145488    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145491    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145500    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145503    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145505    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145510    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145514    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145517    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.145525 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145520    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145523    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145525    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145528    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145531    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145534    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145536    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145540    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145542    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145545    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145548    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145550    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145553    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145555    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145558    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145561    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145563    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145566    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145568    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145571    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.145990 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145573    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145577    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145581    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145584    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145588    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145591    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145594    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145597    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145600    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145603    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145606    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145609    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145612    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145614    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145618    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145621    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145624    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145627    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145629    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145632    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.146483 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145635    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145649    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145652    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145655    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145658    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145661    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145664    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145667    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145670    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145672    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145675    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145678    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145680    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145683    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145685    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145688    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145690    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145693    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145698    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.146981 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145701    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145708    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145711    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145714    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145717    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145719    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145724    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145727    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145729    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145732    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145735    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145738    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145741    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145743    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145746    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145749    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145751    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145754    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.145757    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.147464 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146456    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146461    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146464    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146467    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146470    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146473    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146477    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146480    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146484    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146487    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146490    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146493    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146496    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146498    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146502    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146505    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146507    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146510    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146512    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.148017 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146516    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146519    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146522    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146524    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146527    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146530    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146533    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146536    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146539    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146541    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146544    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146547    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146549    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146552    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146555    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146557    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146560    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146562    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146565    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146567    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.148625 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146571    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146575    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146579    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146581    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146584    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146587    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146590    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146592    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146595    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146598    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146601    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146604    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146606    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146609    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146611    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146614    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146616    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146619    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146621    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146624    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.149268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146626    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146629    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146631    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146634    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146636    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146655    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146660    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146663    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146670    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146673    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146676    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146678    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146681    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146684    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146688    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146691    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146695    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146698    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146700    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.149987 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146703    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146706    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146709    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146712    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146714    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146717    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146720    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146722    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146724    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146727    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146729    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146732    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146735    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146737    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146740    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146742    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146745    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146748    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146750    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146753    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.150477 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.146756    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147134    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147139    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147141    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147144    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147150    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147153    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147156    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147159    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147161    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147164    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147166    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147169    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147172    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147178    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147181    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147183    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147186    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147188    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147191    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.150989 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147194    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147197    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147200    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147203    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147205    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147208    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147211    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147213    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147215    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147218    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147220    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147223    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147228    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147230    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147233    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147235    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147238    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147241    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147245    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.151478 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147249    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147252    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147255    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147258    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147262    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147264    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147267    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147269    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147272    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147275    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147278    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147281    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147283    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147286    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147288    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147291    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147293    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147296    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147298    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147301    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.151961 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147305    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147307    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147310    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147312    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147314    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147318    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147321    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147324    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147326    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147330    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147335    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147337    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147340    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147343    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147345    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147348    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147351    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147353    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147356    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.152441 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147358    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147361    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147364    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147366    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147369    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147372    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147374    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147377    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147379    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147382    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147384    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147387    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147389    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147392    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147394    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147396    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147399    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147401    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147405    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147407    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.152970 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147410    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.147412    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147491    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147532    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147540    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147545    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147549    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147553    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147557    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147561    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147564    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147567    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147570    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147574    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147577    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147581    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147584    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147587    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147590    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147592    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147597    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147600    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147603    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147606    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:41:03.153509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147609    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147613    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147616    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147619    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147622    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147625    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147630    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147633    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147636    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147653    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147657    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147661    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147664    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147667    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147670    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147673    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147678    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147681    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147684    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147687    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147690    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147693    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147696    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147699    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147703    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:41:03.154100 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147706    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147709    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147712    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147714    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147717    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147720    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147723    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147727    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147730    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147733    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147736    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147739    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147742    2581 flags.go:64] FLAG: --help="false"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147746    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147750    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147753    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147756    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147759    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147764    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147767    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147770    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147773    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147776    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:41:03.154735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147779    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147782    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147785    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147788    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147791    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147794    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147797    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147800    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147803    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147806    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147809    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147817    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147820    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147823    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147826    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147829    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147833    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147835    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147838    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147842    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147847    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147851    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147855    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147858    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147861    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:41:03.155330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147864    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147867    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147871    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147874    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147882    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147885    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147888    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147891    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147894    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147899    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147902    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147905    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147908    2581 flags.go:64] FLAG: --port="10250"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147911    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147914    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09590f5edfaf6e997"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147917    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147920    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147924    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147926    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147930    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147959    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147963    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147966    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147971    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147974    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:41:03.155984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147977    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147980    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147983    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147987    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147991    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147994    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.147997    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148000    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148003    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148006    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148011    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148014    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148017    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148020    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148023    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148026    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148029    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148032    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148038    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148041    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148044    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148048    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148051    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148054    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148057    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:41:03.156607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148061    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148064    2581 flags.go:64] FLAG: --v="2"
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148068    2581 flags.go:64] FLAG: --version="false"
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148072    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148075    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.148079    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148890    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148904    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148910    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148915    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148919    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148926    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148931    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148935    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148939    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148950    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148955    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148960    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148964    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148968    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148972    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.157274 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148976    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148981    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148985    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148990    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148994    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.148999    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149005    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149016    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149021    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149026    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149031    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149036    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149051    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149056    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149060    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149064    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149068    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149072    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149079    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.157815 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149089    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149093    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149098    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149104    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149108    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149114    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149119    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149124    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149128    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149133    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149137    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149142    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149151    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149155    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149159    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149164    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149167    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149172    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149176    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149180    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.158278 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149185    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149189    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149193    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149198    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149202    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149210    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149215    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149220    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149224    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149228    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149233    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149237    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149242    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149246    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149250    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149256    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149260    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149269    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149274    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149278    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.158804 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149282    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149287    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149291    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149295    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149300    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149304    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149308    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149313    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149317    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149322    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149331    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149336    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149340    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149345    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149350    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149354    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149358    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149362    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149366    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149371    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.159297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149375    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149380    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149389    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149393    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.149397    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.150265    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.157884    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.157901    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.157921    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.157930    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.157986    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.157990    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.157994    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.157998    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.159807 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158002    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158006    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158009    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158012    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158015    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158017    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158020    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158023    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158025    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158028    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158031    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158034    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158036    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158039    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158043    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158047    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158050    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158052    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158056    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.160203 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158059    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158062    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158064    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158067    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158069    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158072    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158076    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158078    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158081    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158083    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158086    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158089    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158092    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158094    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158097    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158099    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158102    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158104    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158107    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158109    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.160763 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158112    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158115    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158117    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158120    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158122    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158125    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158128    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158130    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158133    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158136    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158139    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158141    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158144    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158146    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158149    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158151    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158154    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158156    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158160    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158163    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.161260 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158165    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158168    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158170    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158173    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158175    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158178    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158181    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158183    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158186    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158188    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158191    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158194    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158196    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158200    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158204    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158207    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158209    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158213    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158216    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.161772 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158218    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158221    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158223    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158226    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158228    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158231    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158233    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158236    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158238    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158241    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158243    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158247    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158250    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158252    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158255    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158258    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.162239 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158261    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.158266    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158381    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158387    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158390    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158393    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158396    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158399    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158402    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158404    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158407    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158409    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158412    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158415    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158418    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:03.162626 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158421    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158423    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158426    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158429    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158431    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158434    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158437    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158439    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158442    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158444    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158447    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158449    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158452    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158455    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158458    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158461    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158463    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158466    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158469    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:03.163038 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158471    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158473    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158476    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158478    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158481    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158484    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158487    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158489    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158491    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158494    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158497    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158499    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158503    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158507    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158511    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158514    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158517    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158521    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158524    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:03.163498 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158526    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158529    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158532    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158535    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158537    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158540    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158542    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158545    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158548    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158551    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158553    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158556    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158558    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158561    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158564    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158566    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158568    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158571    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158574    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158576    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:03.163975 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158579    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158581    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158584    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158586    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158590    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158593    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158597    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158600    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158603    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158606    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158609    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158611    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158614    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158617    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158619    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158622    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158624    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158626    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158629    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:03.164459 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158631    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158634    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158636    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158655    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158658    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158661    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158663    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158666    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:03.158669    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.158675    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.158797    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.160952    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.161862    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.161960    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:03.164943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.162490    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:03.165304 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.164180    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:03.165340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.165322    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:03.169949 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.169733    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:41:03.172080 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.172063    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 17:41:03.172863 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.172847    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:41:03.175430 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.175399    2581 fs.go:135] Filesystem UUIDs: map[1a023873-ba1a-4602-950f-b3159fa0076a:/dev/nvme0n1p4 25cf25a5-e283-44d8-869a-5e97e8e1598c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:41:03.175533 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.175428    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:41:03.181828 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.181671    2581 manager.go:216] Machine: {Timestamp:2026-09-17 17:41:03.1811585 +0000 UTC m=+2.049291454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126157 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cf35f9a2761f531ad56897064120f SystemUUID:ec2cf35f-9a27-61f5-31ad-56897064120f BootID:3f95f49d-e605-4eae-9601-0d7accd1c8d5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:17:aa:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:17:aa:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:2c:a2:c0:f6:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:41:03.181828 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.181807    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:41:03.181994 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.181912    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:41:03.183373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.183335    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:41:03.183510 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.183381    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:41:03.184592 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.184582    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:41:03.184632 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.184596    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:41:03.184632 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.184603    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:03.184632 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.184614    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:03.185849 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.185838    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:41:03.188805 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188793    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:03.188908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188900    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:41:03.188940 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188932    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:41:03.188968 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188942    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:41:03.188968 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188956    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:41:03.188968 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188965    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:41:03.189073 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.188973    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:41:03.189320 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.189307    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:03.189368 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.189329    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:03.191407 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.191390    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:41:03.194569 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194553    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:41:03.194668 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194584    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:41:03.194778 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194767    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:41:03.194811 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194783    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:41:03.194811 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194790    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:41:03.194811 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194795    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:41:03.194811 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194803    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:41:03.194811 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.194809    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:41:03.195221 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195212    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:41:03.195221 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195221    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:41:03.195280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195229    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:41:03.195280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195234    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:41:03.195280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195251    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:41:03.195280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195259    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:41:03.195280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195282    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:41:03.195431 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.195287    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:41:03.198094 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.198074    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:03.199447 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.199432    2581 server.go:1268] "Started kubelet"
Sep 17 17:41:03.199542 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.199514    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:41:03.199686 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.199608    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:41:03.199756 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.199744    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:41:03.200105 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.200079    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:41:03.200731 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.200695    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:41:03.200731 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.200695    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:41:03.200875 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.200801    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:41:03.201190 ip-10-0-134-156 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:41:03.202391 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.202268    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:41:03.204091 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.204066    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:41:03.208872 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.208854    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:41:03.208872 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.208861    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:03.209540 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.209269    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.209540 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.209426    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:41:03.209540 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.209439    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:41:03.209540 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.209452    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:41:03.228602 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.228579    2581 factory.go:55] Registering systemd factory
Sep 17 17:41:03.228602 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.228607    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:41:03.228917 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.228823    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:41:03.229001 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.228954    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:41:03.229001 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.228964    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:41:03.229001 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.228942    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:41:03.229123 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.229096    2581 factory.go:153] Registering CRI-O factory
Sep 17 17:41:03.229123 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.229111    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:41:03.229256 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.229227    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:41:03.229347 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.229274    2581 factory.go:103] Registering Raw factory
Sep 17 17:41:03.229347 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.229300    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:41:03.230130 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.230110    2581 manager.go:349] Starting recovery of all containers
Sep 17 17:41:03.232467 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.232414    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qt528"
Sep 17 17:41:03.240746 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.240570    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qt528"
Sep 17 17:41:03.241523 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.241509    2581 manager.go:354] Recovery completed
Sep 17 17:41:03.243780 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.242746    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18d62c87af49c686  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-09-17 17:41:03.199405702 +0000 UTC m=+2.067538656,LastTimestamp:2026-09-17 17:41:03.199405702 +0000 UTC m=+2.067538656,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}"
Sep 17 17:41:03.245843 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.245828    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.247833 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.247817    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.247902 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.247857    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.247902 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.247874    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.248453 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.248436    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:41:03.248453 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.248449    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:41:03.248585 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.248462    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:03.248585 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.248486    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.249270 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249255    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.249332 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249281    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.249332 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249294    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.249614 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249599    2581 policy_none.go:49] "None policy: Start"
Sep 17 17:41:03.249614 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249616    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:41:03.249743 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.249627    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:41:03.250742 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.250731    2581 policy_none.go:47] "Start"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.290867    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.290905    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.290916    2581 server.go:87] "Starting device plugin registration server"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.291178    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.291203    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.291318    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.291408    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.291418    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.291990    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:41:03.293343 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.292033    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.354054 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.354017    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:41:03.355200 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.355183    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:41:03.355258 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.355231    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:41:03.355258 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.355251    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:41:03.355332 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.355285    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:41:03.357330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.357310    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.392128 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.392052    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.393087 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.393070    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.393193 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.393116    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.393193 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.393134    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.393193 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.393160    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.401331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.401315    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.401415 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.401343    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-156.ec2.internal\": node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.420727 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.420706    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.456082 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.456049    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"]
Sep 17 17:41:03.456177 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.456124    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.457058 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.457046    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.457143 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.457069    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.457143 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.457079    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.458509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.458495    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.458694 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.458680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.458768 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.458738    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.459248 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459220    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.459248 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459244    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.459340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459254    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.459340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459264    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.459340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459284    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.459340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.459303    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.460594 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.460580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.460654 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.460607    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.461544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.461527    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.461602 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.461562    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.461602 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.461580    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.484316 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.484298    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.488891 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.488873    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.521707 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.521688    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.530920 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.530897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/500d1624d8f41e3c56732fe6161a664f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"500d1624d8f41e3c56732fe6161a664f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.531000 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.530924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.531000 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.530945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.622637 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.622604    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.631836 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/500d1624d8f41e3c56732fe6161a664f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"500d1624d8f41e3c56732fe6161a664f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.631836 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/500d1624d8f41e3c56732fe6161a664f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"500d1624d8f41e3c56732fe6161a664f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.632023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.632023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.632023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.632023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.631965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257db8efb146f5ee70a4d9e0d9c4c99c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"257db8efb146f5ee70a4d9e0d9c4c99c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.723535 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.723453    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.787780 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.787759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.793163 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:03.793142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:03.823738 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.823707    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:03.924204 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:03.924164    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:04.024697 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.024618    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:04.125224 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.125202    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found"
Sep 17 17:41:04.143936 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.143918    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:04.161873 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.161851    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:41:04.161977 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.161955    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:04.162029 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.161996    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:04.174905 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.174885    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:04.189574 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.189561    2581 apiserver.go:52] "Watching apiserver"
Sep 17 17:41:04.198547 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.198529    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:41:04.198890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.198868    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-mhwgg","openshift-image-registry/node-ca-8gjbf","openshift-multus/multus-additional-cni-plugins-nmzvx","openshift-multus/multus-vsfpn","openshift-multus/network-metrics-daemon-4cbwc","kube-system/global-pull-secret-syncer-5rspk","openshift-network-diagnostics/network-check-target-lfzxt","openshift-network-operator/iptables-alerter-jvzls","openshift-ovn-kubernetes/ovnkube-node-g6ksl","kube-system/konnectivity-agent-lpl5h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r","openshift-cluster-node-tuning-operator/tuned-j255r"]
Sep 17 17:41:04.200649 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.200624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.201818 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.201803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.202893 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.202880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.203541 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.203526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.203618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.203579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dmw27\""
Sep 17 17:41:04.203678 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.203664    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.204040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.204027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.204630 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.204615    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.204708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.204690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6l7ct\""
Sep 17 17:41:04.204883 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.204869    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:41:04.205052 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205040    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.205428 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.205502 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.205483    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:04.205786 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.205854 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:41:04.205914 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:41:04.205914 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:41:04.205999 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.205851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flss9\""
Sep 17 17:41:04.206186 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.206171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.206908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.206893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.206971 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.206953    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:04.207697 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.207683    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wfmdv\""
Sep 17 17:41:04.207766 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.207693    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:41:04.208008 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.207995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:04.208054 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.208039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:04.209324 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.209311    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:04.209399 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.209313    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:04.209788 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.209770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.211088 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.211073    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.212898 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.212876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.215021 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.215003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.216450 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.216431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.218340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.218323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.218623 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.218606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.218729 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.218628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:41:04.218729 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.218603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5nz9z\""
Sep 17 17:41:04.219033 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.219018    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:41:04.219085 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.219077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.221806 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.221785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xsvs4\""
Sep 17 17:41:04.221894 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.221862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gvbv4\""
Sep 17 17:41:04.221894 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.221874    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:41:04.221991 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.221908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:41:04.221991 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.221952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:41:04.222088 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:41:04.222213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nb8hq\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222250    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxzvp\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222287    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.222307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:41:04.222541 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.222361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:41:04.234444 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cnibin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-log-socket\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.234543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-config\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.234543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-env-overrides\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.234543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cni-binary-copy\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-daemon-config\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4tr\" (UniqueName: \"kubernetes.io/projected/1ff9e614-f637-4494-a951-05bc50d222e2-kube-api-access-vx4tr\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-etc-kubernetes\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-etc-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-netns\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-kubelet\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.234785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/47daedcd-8541-40ae-847e-7b786c4a982b-kube-api-access-j8sr4\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vrq\" (UniqueName: \"kubernetes.io/projected/8215753a-c993-4f2a-82d1-756ab30df519-kube-api-access-k5vrq\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-dbus\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-systemd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-ovn\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.234979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-netd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-bin\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-k8s-cni-cncf-io\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ff9e614-f637-4494-a951-05bc50d222e2-iptables-alerter-script\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.235110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cnibin\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-systemd-units\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-multus\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47daedcd-8541-40ae-847e-7b786c4a982b-serviceca\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-kubelet-config\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db7e86f5-234c-409d-96c1-a57c43db924e-ovn-node-metrics-cert\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-bin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdg9\" (UniqueName: \"kubernetes.io/projected/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-kube-api-access-wrdg9\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4ds\" (UniqueName: \"kubernetes.io/projected/2a4a871b-b563-4ba1-b00e-d0687edf4d03-kube-api-access-lh4ds\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-netns\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-script-lib\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-conf-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.235666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ff9e614-f637-4494-a951-05bc50d222e2-host-slash\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-slash\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3029b88-7089-4632-bd93-a6323afb9b49-tmp-dir\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47daedcd-8541-40ae-847e-7b786c4a982b-host\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-node-log\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sdr\" (UniqueName: \"kubernetes.io/projected/c3029b88-7089-4632-bd93-a6323afb9b49-kube-api-access-82sdr\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-system-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-hostroot\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-multus-certs\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-system-cni-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-kubelet\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-var-lib-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3029b88-7089-4632-bd93-a6323afb9b49-hosts-file\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.235959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-socket-dir-parent\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.236012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.236271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.236056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9ps\" (UniqueName: \"kubernetes.io/projected/db7e86f5-234c-409d-96c1-a57c43db924e-kube-api-access-jr9ps\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.242996 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.242980    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:04.244557 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.244495    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:36:03 +0000 UTC" deadline="2028-06-24 11:01:04.60101714 +0000 UTC"
Sep 17 17:41:04.244557 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.244557    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15497h20m0.35646424s"
Sep 17 17:41:04.252181 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.252164    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:04.252264 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.252252    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"
Sep 17 17:41:04.252373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.252357    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal"]
Sep 17 17:41:04.274769 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.274722    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"]
Sep 17 17:41:04.277138 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.277124    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:04.295881 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.295857    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jtfm"
Sep 17 17:41:04.308438 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.308418    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jtfm"
Sep 17 17:41:04.310601 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.310581    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:41:04.337171 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-etc-kubernetes\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-etc-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-kubernetes\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.337279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-var-lib-kubelet\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.337279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-tmp\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.337279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-netns\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-etc-kubernetes\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-etc-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-kubelet\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-netns\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-kubelet\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-sys-fs\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/47daedcd-8541-40ae-847e-7b786c4a982b-kube-api-access-j8sr4\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.337479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vrq\" (UniqueName: \"kubernetes.io/projected/8215753a-c993-4f2a-82d1-756ab30df519-kube-api-access-k5vrq\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-dbus\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-dbus\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-systemd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-ovn\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-netd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysconfig\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-systemd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4157521a-db9f-46a3-922e-afd22e39edb3-konnectivity-ca\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-ovn\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-netd\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-bin\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-k8s-cni-cncf-io\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.337979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ff9e614-f637-4494-a951-05bc50d222e2-iptables-alerter-script\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cnibin\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-cni-bin\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-k8s-cni-cncf-io\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-systemd-units\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.337977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-registration-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-multus\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47daedcd-8541-40ae-847e-7b786c4a982b-serviceca\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46x78\" (UniqueName: \"kubernetes.io/projected/6471513d-3e04-421b-8d90-7005d966cefc-kube-api-access-46x78\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-systemd-units\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-tuned\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-multus\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-kubelet-config\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.338822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-kubelet-config\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db7e86f5-234c-409d-96c1-a57c43db924e-ovn-node-metrics-cert\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-device-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cnibin\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-sys\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-bin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a4a871b-b563-4ba1-b00e-d0687edf4d03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdg9\" (UniqueName: \"kubernetes.io/projected/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-kube-api-access-wrdg9\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ff9e614-f637-4494-a951-05bc50d222e2-iptables-alerter-script\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-host\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4ds\" (UniqueName: \"kubernetes.io/projected/2a4a871b-b563-4ba1-b00e-d0687edf4d03-kube-api-access-lh4ds\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-var-lib-cni-bin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-netns\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47daedcd-8541-40ae-847e-7b786c4a982b-serviceca\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338621    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:41:04.339468 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-run-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-run-netns\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-script-lib\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-conf-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ff9e614-f637-4494-a951-05bc50d222e2-host-slash\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ff9e614-f637-4494-a951-05bc50d222e2-host-slash\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-conf-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-slash\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3029b88-7089-4632-bd93-a6323afb9b49-tmp-dir\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-conf\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-slash\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.338998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-systemd\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47daedcd-8541-40ae-847e-7b786c4a982b-host\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.340196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-node-log\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47daedcd-8541-40ae-847e-7b786c4a982b-host\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sdr\" (UniqueName: \"kubernetes.io/projected/c3029b88-7089-4632-bd93-a6323afb9b49-kube-api-access-82sdr\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.339160    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-node-log\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3029b88-7089-4632-bd93-a6323afb9b49-tmp-dir\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-lib-modules\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.339242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.839222463 +0000 UTC m=+3.707355424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtsz\" (UniqueName: \"kubernetes.io/projected/18115ca3-a167-44dd-b0bf-c73261e8881c-kube-api-access-jgtsz\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-system-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-script-lib\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-hostroot\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-system-cni-dir\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-multus-certs\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-host-run-multus-certs\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.340868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-system-cni-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-hostroot\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-kubelet\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-var-lib-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-host-kubelet\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3029b88-7089-4632-bd93-a6323afb9b49-hosts-file\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-var-lib-openvswitch\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-socket-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a4a871b-b563-4ba1-b00e-d0687edf4d03-system-cni-dir\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3029b88-7089-4632-bd93-a6323afb9b49-hosts-file\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-socket-dir-parent\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-socket-dir-parent\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9ps\" (UniqueName: \"kubernetes.io/projected/db7e86f5-234c-409d-96c1-a57c43db924e-kube-api-access-jr9ps\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-run\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.339637    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4157521a-db9f-46a3-922e-afd22e39edb3-agent-certs\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.341376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cnibin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.339742    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.839725286 +0000 UTC m=+3.707858248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cnibin\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-log-socket\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-config\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-env-overrides\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-modprobe-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cni-binary-copy\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-daemon-config\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db7e86f5-234c-409d-96c1-a57c43db924e-log-socket\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.339973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4tr\" (UniqueName: \"kubernetes.io/projected/1ff9e614-f637-4494-a951-05bc50d222e2-kube-api-access-vx4tr\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.340302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-ovnkube-config\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.340299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db7e86f5-234c-409d-96c1-a57c43db924e-env-overrides\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.340391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-cni-binary-copy\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.340437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-multus-daemon-config\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.341911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.341666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db7e86f5-234c-409d-96c1-a57c43db924e-ovn-node-metrics-cert\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.346578 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.346560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/47daedcd-8541-40ae-847e-7b786c4a982b-kube-api-access-j8sr4\") pod \"node-ca-8gjbf\" (UID: \"47daedcd-8541-40ae-847e-7b786c4a982b\") " pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.346809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.346794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vrq\" (UniqueName: \"kubernetes.io/projected/8215753a-c993-4f2a-82d1-756ab30df519-kube-api-access-k5vrq\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.349911 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.349893    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:04.349911 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.349912    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:04.350019 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.349921    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.350019 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.349966    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.849952848 +0000 UTC m=+3.718085792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.353196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.353174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4ds\" (UniqueName: \"kubernetes.io/projected/2a4a871b-b563-4ba1-b00e-d0687edf4d03-kube-api-access-lh4ds\") pod \"multus-additional-cni-plugins-nmzvx\" (UID: \"2a4a871b-b563-4ba1-b00e-d0687edf4d03\") " pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.353509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.353471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4tr\" (UniqueName: \"kubernetes.io/projected/1ff9e614-f637-4494-a951-05bc50d222e2-kube-api-access-vx4tr\") pod \"iptables-alerter-jvzls\" (UID: \"1ff9e614-f637-4494-a951-05bc50d222e2\") " pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.353656 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.353625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdg9\" (UniqueName: \"kubernetes.io/projected/ad99e04b-e33b-432d-bb24-dc9da16c2f3f-kube-api-access-wrdg9\") pod \"multus-vsfpn\" (UID: \"ad99e04b-e33b-432d-bb24-dc9da16c2f3f\") " pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.353896 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.353880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sdr\" (UniqueName: \"kubernetes.io/projected/c3029b88-7089-4632-bd93-a6323afb9b49-kube-api-access-82sdr\") pod \"node-resolver-mhwgg\" (UID: \"c3029b88-7089-4632-bd93-a6323afb9b49\") " pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.353941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.353876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9ps\" (UniqueName: \"kubernetes.io/projected/db7e86f5-234c-409d-96c1-a57c43db924e-kube-api-access-jr9ps\") pod \"ovnkube-node-g6ksl\" (UID: \"db7e86f5-234c-409d-96c1-a57c43db924e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.435053 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.435008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d1624d8f41e3c56732fe6161a664f.slice/crio-ee77144f955b4a88c66e17bae3157ed7d40ea51aa4b5b375fe0dadbcdfb9ff3c WatchSource:0}: Error finding container ee77144f955b4a88c66e17bae3157ed7d40ea51aa4b5b375fe0dadbcdfb9ff3c: Status 404 returned error can't find the container with id ee77144f955b4a88c66e17bae3157ed7d40ea51aa4b5b375fe0dadbcdfb9ff3c
Sep 17 17:41:04.435440 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.435415    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257db8efb146f5ee70a4d9e0d9c4c99c.slice/crio-7111adf589f087ca43ae5e18c8352e0708ec83127f9e4fd22c25696e7d9159f2 WatchSource:0}: Error finding container 7111adf589f087ca43ae5e18c8352e0708ec83127f9e4fd22c25696e7d9159f2: Status 404 returned error can't find the container with id 7111adf589f087ca43ae5e18c8352e0708ec83127f9e4fd22c25696e7d9159f2
Sep 17 17:41:04.440267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440247    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:04.440367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-registration-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46x78\" (UniqueName: \"kubernetes.io/projected/6471513d-3e04-421b-8d90-7005d966cefc-kube-api-access-46x78\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-tuned\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-etc-selinux\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-registration-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-device-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-sys\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-device-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-host\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-sys\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.440570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-conf\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-systemd\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-host\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-lib-modules\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtsz\" (UniqueName: \"kubernetes.io/projected/18115ca3-a167-44dd-b0bf-c73261e8881c-kube-api-access-jgtsz\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-conf\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-socket-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-systemd\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-run\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-lib-modules\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-socket-dir\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4157521a-db9f-46a3-922e-afd22e39edb3-agent-certs\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-modprobe-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-kubernetes\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-var-lib-kubelet\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-tmp\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-sys-fs\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-run\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysconfig\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4157521a-db9f-46a3-922e-afd22e39edb3-konnectivity-ca\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-kubernetes\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-var-lib-kubelet\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-modprobe-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.440966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysctl-d\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.441011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6471513d-3e04-421b-8d90-7005d966cefc-sys-fs\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.441018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-sysconfig\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.441823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.441437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4157521a-db9f-46a3-922e-afd22e39edb3-konnectivity-ca\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.442545 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.442526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-etc-tuned\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.442991 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.442971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4157521a-db9f-46a3-922e-afd22e39edb3-agent-certs\") pod \"konnectivity-agent-lpl5h\" (UID: \"4157521a-db9f-46a3-922e-afd22e39edb3\") " pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.443110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.443092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18115ca3-a167-44dd-b0bf-c73261e8881c-tmp\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.448779 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.448758    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:04.449480 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.449465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46x78\" (UniqueName: \"kubernetes.io/projected/6471513d-3e04-421b-8d90-7005d966cefc-kube-api-access-46x78\") pod \"aws-ebs-csi-driver-node-vk99r\" (UID: \"6471513d-3e04-421b-8d90-7005d966cefc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.456362 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.456324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtsz\" (UniqueName: \"kubernetes.io/projected/18115ca3-a167-44dd-b0bf-c73261e8881c-kube-api-access-jgtsz\") pod \"tuned-j255r\" (UID: \"18115ca3-a167-44dd-b0bf-c73261e8881c\") " pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.530869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.530850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhwgg"
Sep 17 17:41:04.538985 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.538968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gjbf"
Sep 17 17:41:04.539350 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.539331    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3029b88_7089_4632_bd93_a6323afb9b49.slice/crio-b98c1aeb848aca330aa0e7a8a455504b24086fe98183f30c7ad1c422a756cd4f WatchSource:0}: Error finding container b98c1aeb848aca330aa0e7a8a455504b24086fe98183f30c7ad1c422a756cd4f: Status 404 returned error can't find the container with id b98c1aeb848aca330aa0e7a8a455504b24086fe98183f30c7ad1c422a756cd4f
Sep 17 17:41:04.545942 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.545923    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47daedcd_8541_40ae_847e_7b786c4a982b.slice/crio-85b6631ea9c1472cbc1961dae6a49278e05ae5d0268f3cdcbb773a9ea295bf25 WatchSource:0}: Error finding container 85b6631ea9c1472cbc1961dae6a49278e05ae5d0268f3cdcbb773a9ea295bf25: Status 404 returned error can't find the container with id 85b6631ea9c1472cbc1961dae6a49278e05ae5d0268f3cdcbb773a9ea295bf25
Sep 17 17:41:04.555711 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.555692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmzvx"
Sep 17 17:41:04.564297 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.564278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4a871b_b563_4ba1_b00e_d0687edf4d03.slice/crio-e81eaeeb7f718d56557a714266285761f849c5b46cdbb4db05f71a204172a0bf WatchSource:0}: Error finding container e81eaeeb7f718d56557a714266285761f849c5b46cdbb4db05f71a204172a0bf: Status 404 returned error can't find the container with id e81eaeeb7f718d56557a714266285761f849c5b46cdbb4db05f71a204172a0bf
Sep 17 17:41:04.567278 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.567257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsfpn"
Sep 17 17:41:04.573892 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.573877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad99e04b_e33b_432d_bb24_dc9da16c2f3f.slice/crio-8b8f1e593daf34f44e5ff462993018398115bde5cc559982974da93b3faef5d1 WatchSource:0}: Error finding container 8b8f1e593daf34f44e5ff462993018398115bde5cc559982974da93b3faef5d1: Status 404 returned error can't find the container with id 8b8f1e593daf34f44e5ff462993018398115bde5cc559982974da93b3faef5d1
Sep 17 17:41:04.582703 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.582687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvzls"
Sep 17 17:41:04.588908 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.588889    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff9e614_f637_4494_a951_05bc50d222e2.slice/crio-9308ef95afbabc59939f64477dba6ade97b8d1836a32f55ced66d3cd0b63b403 WatchSource:0}: Error finding container 9308ef95afbabc59939f64477dba6ade97b8d1836a32f55ced66d3cd0b63b403: Status 404 returned error can't find the container with id 9308ef95afbabc59939f64477dba6ade97b8d1836a32f55ced66d3cd0b63b403
Sep 17 17:41:04.597243 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.597228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:04.603553 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.603535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7e86f5_234c_409d_96c1_a57c43db924e.slice/crio-eef49206aa7c8f4ae36d814a909617de5a3349c17ce49d4b495a54d8c7dd26ee WatchSource:0}: Error finding container eef49206aa7c8f4ae36d814a909617de5a3349c17ce49d4b495a54d8c7dd26ee: Status 404 returned error can't find the container with id eef49206aa7c8f4ae36d814a909617de5a3349c17ce49d4b495a54d8c7dd26ee
Sep 17 17:41:04.611504 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.611488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:04.616725 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.616709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r"
Sep 17 17:41:04.618019 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.617998    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4157521a_db9f_46a3_922e_afd22e39edb3.slice/crio-4f647346e200e66188191f0acc23442cd4b5c21ae5dfefc9f7d22cfa740a0fc3 WatchSource:0}: Error finding container 4f647346e200e66188191f0acc23442cd4b5c21ae5dfefc9f7d22cfa740a0fc3: Status 404 returned error can't find the container with id 4f647346e200e66188191f0acc23442cd4b5c21ae5dfefc9f7d22cfa740a0fc3
Sep 17 17:41:04.623206 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:04.623187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6471513d_3e04_421b_8d90_7005d966cefc.slice/crio-3dd506dee3d0e94e1a9ee1578fdebf86130d1d81291856fcecec67c2f02e0f4a WatchSource:0}: Error finding container 3dd506dee3d0e94e1a9ee1578fdebf86130d1d81291856fcecec67c2f02e0f4a: Status 404 returned error can't find the container with id 3dd506dee3d0e94e1a9ee1578fdebf86130d1d81291856fcecec67c2f02e0f4a
Sep 17 17:41:04.632399 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.632378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j255r"
Sep 17 17:41:04.845111 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.845015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:04.845111 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.845074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:04.845277 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.845171    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.845277 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.845236    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.845217293 +0000 UTC m=+4.713350248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.845277 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.845172    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.845412 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.845310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.845297829 +0000 UTC m=+4.713430769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.946012 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:04.945975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:04.946196 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.946161    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:04.946196 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.946195    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:04.946325 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.946210    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.946325 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:04.946273    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.946252576 +0000 UTC m=+4.814385528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.024855 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.024823    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:05.311551 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.311491    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:04 +0000 UTC" deadline="2028-05-17 12:59:22.358723574 +0000 UTC"
Sep 17 17:41:05.311551 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.311550    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14587h18m17.047177337s"
Sep 17 17:41:05.356614 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.356579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:05.356787 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.356732    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:05.382849 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.382760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"257db8efb146f5ee70a4d9e0d9c4c99c","Type":"ContainerStarted","Data":"7111adf589f087ca43ae5e18c8352e0708ec83127f9e4fd22c25696e7d9159f2"}
Sep 17 17:41:05.408745 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.408710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsfpn" event={"ID":"ad99e04b-e33b-432d-bb24-dc9da16c2f3f","Type":"ContainerStarted","Data":"8b8f1e593daf34f44e5ff462993018398115bde5cc559982974da93b3faef5d1"}
Sep 17 17:41:05.430207 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.430174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gjbf" event={"ID":"47daedcd-8541-40ae-847e-7b786c4a982b","Type":"ContainerStarted","Data":"85b6631ea9c1472cbc1961dae6a49278e05ae5d0268f3cdcbb773a9ea295bf25"}
Sep 17 17:41:05.439505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.439475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhwgg" event={"ID":"c3029b88-7089-4632-bd93-a6323afb9b49","Type":"ContainerStarted","Data":"b98c1aeb848aca330aa0e7a8a455504b24086fe98183f30c7ad1c422a756cd4f"}
Sep 17 17:41:05.449939 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.449872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j255r" event={"ID":"18115ca3-a167-44dd-b0bf-c73261e8881c","Type":"ContainerStarted","Data":"ec4cff2c4619fcb5d9e7253cd18faf93963dfc5fc5b83663629a7a6b40fbdb41"}
Sep 17 17:41:05.468248 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.468218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r" event={"ID":"6471513d-3e04-421b-8d90-7005d966cefc","Type":"ContainerStarted","Data":"3dd506dee3d0e94e1a9ee1578fdebf86130d1d81291856fcecec67c2f02e0f4a"}
Sep 17 17:41:05.480802 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.480765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpl5h" event={"ID":"4157521a-db9f-46a3-922e-afd22e39edb3","Type":"ContainerStarted","Data":"4f647346e200e66188191f0acc23442cd4b5c21ae5dfefc9f7d22cfa740a0fc3"}
Sep 17 17:41:05.498040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.497913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"eef49206aa7c8f4ae36d814a909617de5a3349c17ce49d4b495a54d8c7dd26ee"}
Sep 17 17:41:05.508662 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.508588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvzls" event={"ID":"1ff9e614-f637-4494-a951-05bc50d222e2","Type":"ContainerStarted","Data":"9308ef95afbabc59939f64477dba6ade97b8d1836a32f55ced66d3cd0b63b403"}
Sep 17 17:41:05.534325 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.534290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerStarted","Data":"e81eaeeb7f718d56557a714266285761f849c5b46cdbb4db05f71a204172a0bf"}
Sep 17 17:41:05.565602 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.565525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"500d1624d8f41e3c56732fe6161a664f","Type":"ContainerStarted","Data":"ee77144f955b4a88c66e17bae3157ed7d40ea51aa4b5b375fe0dadbcdfb9ff3c"}
Sep 17 17:41:05.854890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.854801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:05.854890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.854858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:05.855090 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.855006    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:05.855090 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.855073    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.85505311 +0000 UTC m=+6.723186066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:05.855487 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.855453    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:05.855593 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.855504    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.855488495 +0000 UTC m=+6.723621442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:05.955705 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:05.955672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:05.955878 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.955793    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:05.955878 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.955807    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:05.955878 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.955817    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.955878 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:05.955873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.955856199 +0000 UTC m=+6.823989146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:06.311922 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:06.311879    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:04 +0000 UTC" deadline="2028-02-15 11:19:58.88665935 +0000 UTC"
Sep 17 17:41:06.311922 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:06.311920    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12377h38m52.57474298s"
Sep 17 17:41:06.355681 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:06.355634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:06.355856 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:06.355808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:06.355928 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:06.355904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:06.356292 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:06.356269    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:07.329173 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:07.329145    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:07.358597 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:07.358568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:07.358774 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.358718    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:07.871303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:07.871361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.871494    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.871557    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.871538472 +0000 UTC m=+10.739671419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.871990    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:07.872079 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.872041    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.872026645 +0000 UTC m=+10.740159593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:07.973043 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:07.972370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:07.973043 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.972603    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:07.973043 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.972626    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:07.973043 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.972655    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:07.973043 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:07.972710    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.972692848 +0000 UTC m=+10.840825795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:08.356059 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:08.356027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:08.356503 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:08.356082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:08.356503 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:08.356241    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:08.356503 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:08.356408    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:09.355557 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:09.355519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:09.355747 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:09.355671    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:10.356242 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:10.356174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:10.356242 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:10.356197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:10.356796 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:10.356309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:10.356796 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:10.356492    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:11.356089 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:11.355810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:11.356089 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:11.355943    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:11.904954 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:11.904918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:11.904954 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:11.904960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:11.905187 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:11.905085    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.905187 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:11.905149    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:19.905130232 +0000 UTC m=+18.773263177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:11.905187 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:11.905085    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:11.905353 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:11.905212    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:19.905198491 +0000 UTC m=+18.773331436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:12.005726 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:12.005686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:12.005923 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.005906    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:12.005988 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.005937    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:12.005988 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.005953    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.006093 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.006012    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:20.005994902 +0000 UTC m=+18.874127861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.355461 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:12.355431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:12.355667 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.355562    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:12.355952 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:12.355935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:12.356068 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:12.356050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:13.356981 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:13.356942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:13.357413 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:13.357089    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:14.356104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:14.356067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:14.356298 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:14.356070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:14.356298 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:14.356203    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:14.356419 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:14.356308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:15.356147 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:15.356112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:15.356736 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:15.356224    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:16.355716 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:16.355681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:16.355882 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:16.355738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:16.355882 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:16.355805    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:16.356003 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:16.355957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:17.356250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:17.356219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:17.356657 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:17.356346    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:18.356482 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:18.356447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:18.356884 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:18.356596    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:18.356884 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:18.356659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:18.356884 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:18.356752    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:19.356241 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:19.356216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:19.356380 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:19.356311    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:19.963447 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:19.963366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:19.963447 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:19.963414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:19.963963 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:19.963544    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:19.963963 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:19.963601    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:35.963584409 +0000 UTC m=+34.831717350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:19.963963 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:19.963541    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:19.963963 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:19.963714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:41:35.96369613 +0000 UTC m=+34.831829072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:20.063996 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:20.063958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:20.064180 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.064128    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:20.064180 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.064151    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:20.064180 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.064164    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:20.064343 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.064223    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.064204418 +0000 UTC m=+34.932337380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:20.355986 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:20.355951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:20.356242 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:20.356066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:20.356242 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.356105    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:20.356242 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:20.356159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:21.356159 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:21.356122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:21.356621 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:21.356264    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:22.355547 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:22.355515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:22.355750 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:22.355516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:22.355750 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:22.355667    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:22.355750 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:22.355719    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:23.356777 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:23.356746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:23.357086 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:23.356834    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:23.616451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:23.616137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"500d1624d8f41e3c56732fe6161a664f","Type":"ContainerStarted","Data":"6fc1a7dfa98dbfbca12363b81749f1acd884b1afac815f361b60decc75790b9a"}
Sep 17 17:41:24.355983 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.355916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:24.356106 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:24.356012    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:24.356106 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.356066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:24.356223 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:24.356160    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:24.620562 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.620436    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="cc661abfa7016e4140aea0b2466b3f5e8c6e6b8350853251c601ba66c9716655" exitCode=0
Sep 17 17:41:24.620562 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.620530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"cc661abfa7016e4140aea0b2466b3f5e8c6e6b8350853251c601ba66c9716655"}
Sep 17 17:41:24.622363 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.622334    2581 generic.go:356] "Generic (PLEG): container finished" podID="257db8efb146f5ee70a4d9e0d9c4c99c" containerID="4a779b137f869dc09276c9c6f535a5e4c654d4d8ec3a5d976b2c71a3b1e96293" exitCode=0
Sep 17 17:41:24.622460 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.622421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"257db8efb146f5ee70a4d9e0d9c4c99c","Type":"ContainerDied","Data":"4a779b137f869dc09276c9c6f535a5e4c654d4d8ec3a5d976b2c71a3b1e96293"}
Sep 17 17:41:24.624191 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.624164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsfpn" event={"ID":"ad99e04b-e33b-432d-bb24-dc9da16c2f3f","Type":"ContainerStarted","Data":"600dceb81ab98b3d543e3ad34663641b923b08bca770b2a3c6bb439d086b266f"}
Sep 17 17:41:24.625913 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.625884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gjbf" event={"ID":"47daedcd-8541-40ae-847e-7b786c4a982b","Type":"ContainerStarted","Data":"f723ecea83ea4116c39981933a58152233279d5b02f5859dcaf2ad6185c3b504"}
Sep 17 17:41:24.627679 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.627621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhwgg" event={"ID":"c3029b88-7089-4632-bd93-a6323afb9b49","Type":"ContainerStarted","Data":"e9acabfbf779628b35122017b6c5f7c234ff3911b2920bf2a045af099b85237b"}
Sep 17 17:41:24.629251 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.629227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j255r" event={"ID":"18115ca3-a167-44dd-b0bf-c73261e8881c","Type":"ContainerStarted","Data":"f38a76819d4ff6561a2d6362ed7031e533d9269c20b72b3f5fafc38dac2d246f"}
Sep 17 17:41:24.630882 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.630858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r" event={"ID":"6471513d-3e04-421b-8d90-7005d966cefc","Type":"ContainerStarted","Data":"a2d1f8b183c70b9ffc69a1efadd0257111d5518b3a0e017cb61b36d9289d9df4"}
Sep 17 17:41:24.632473 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.632430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpl5h" event={"ID":"4157521a-db9f-46a3-922e-afd22e39edb3","Type":"ContainerStarted","Data":"05afd7eaa0a975139b9d4dc6d2e870c01af937d01aaa5ec563cf9dceb27e3eb8"}
Sep 17 17:41:24.635714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.635693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:41:24.636090 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636066    2581 generic.go:356] "Generic (PLEG): container finished" podID="db7e86f5-234c-409d-96c1-a57c43db924e" containerID="7524ef3de512189c58f091a36dfdc34cc9c782bedfe90ebfda9efd071383be9a" exitCode=1
Sep 17 17:41:24.636243 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"a5985dc652c527af4b3f9529e5fdfc28274a4f8c6e7e3bee3d6d09bd9b44f9c2"}
Sep 17 17:41:24.636243 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"8fe59cb78be3e278799642b6f3c2161ef03900e2cf984ad2dfbd7f45fc16350b"}
Sep 17 17:41:24.636356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"3566b294fa4b71865fb089db63faa0f04b73f21a1dbb6b5382bebd9db1b3e8c5"}
Sep 17 17:41:24.636356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"6f9a5a00d9e7146ceaa6409bee5ac510c273f1ace6e284cfcf09da61dd699ca5"}
Sep 17 17:41:24.636356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerDied","Data":"7524ef3de512189c58f091a36dfdc34cc9c782bedfe90ebfda9efd071383be9a"}
Sep 17 17:41:24.636356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.636341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"e3eff2c0162f4d4a98646711bf8cf26bbc50683e2d749c367a1e2746280d6bb0"}
Sep 17 17:41:24.647828 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.647787    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" podStartSLOduration=20.647775760000002 podStartE2EDuration="20.64777576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:23.633326108 +0000 UTC m=+22.501459072" watchObservedRunningTime="2026-09-17 17:41:24.64777576 +0000 UTC m=+23.515908723"
Sep 17 17:41:24.659714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.659691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:24.660260 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.660246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:24.666624 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.666412    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vsfpn" podStartSLOduration=2.411769875 podStartE2EDuration="21.666396374s" totalImagesPullingTime="19.254626499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.666282802 +0000 UTC m=+23.534415766" watchObservedRunningTime="2026-09-17 17:41:24.666396374 +0000 UTC m=+23.534529339"
Sep 17 17:41:24.683137 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.683098    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lpl5h" podStartSLOduration=2.992323071 podStartE2EDuration="21.683082159s" totalImagesPullingTime="18.690759088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.682013468 +0000 UTC m=+23.550146641" watchObservedRunningTime="2026-09-17 17:41:24.683082159 +0000 UTC m=+23.551215123"
Sep 17 17:41:24.703550 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.703507    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j255r" podStartSLOduration=2.966548678 podStartE2EDuration="21.703494402s" totalImagesPullingTime="18.736945724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.703022471 +0000 UTC m=+23.571155433" watchObservedRunningTime="2026-09-17 17:41:24.703494402 +0000 UTC m=+23.571627364"
Sep 17 17:41:24.745402 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.745361    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mhwgg" podStartSLOduration=2.908754993 podStartE2EDuration="21.745347144s" totalImagesPullingTime="18.836592151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.745272085 +0000 UTC m=+23.613405048" watchObservedRunningTime="2026-09-17 17:41:24.745347144 +0000 UTC m=+23.613480107"
Sep 17 17:41:24.763525 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:24.763485    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8gjbf" podStartSLOduration=2.934303738 podStartE2EDuration="21.763471013s" totalImagesPullingTime="18.829167275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.762877763 +0000 UTC m=+23.631010727" watchObservedRunningTime="2026-09-17 17:41:24.763471013 +0000 UTC m=+23.631603978"
Sep 17 17:41:25.356462 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.356435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:25.356633 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:25.356548    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:25.411475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.411444    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:25.640943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.640865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r" event={"ID":"6471513d-3e04-421b-8d90-7005d966cefc","Type":"ContainerStarted","Data":"bd1769e672f537029bb6f4ada6b9a927112cd5c359f5478e7dca0af088b9f461"}
Sep 17 17:41:25.642679 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.642631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvzls" event={"ID":"1ff9e614-f637-4494-a951-05bc50d222e2","Type":"ContainerStarted","Data":"686aadbb8eebeb74485782e341579a9f24624a68c6b822df354bc95adc82d8bc"}
Sep 17 17:41:25.644725 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.644690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"257db8efb146f5ee70a4d9e0d9c4c99c","Type":"ContainerStarted","Data":"4a107513d80d16374af0291e3b39329ea76deb2d9bf4bf83722af2ed506a145f"}
Sep 17 17:41:25.645059 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.645022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:25.645561 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.645541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lpl5h"
Sep 17 17:41:25.721536 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.721484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jvzls" podStartSLOduration=3.929623574 podStartE2EDuration="22.721467205s" totalImagesPullingTime="18.791843631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:25.675095149 +0000 UTC m=+24.543228117" watchObservedRunningTime="2026-09-17 17:41:25.721467205 +0000 UTC m=+24.589600170"
Sep 17 17:41:25.721745 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:25.721717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" podStartSLOduration=21.721707219 podStartE2EDuration="21.721707219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:25.721014561 +0000 UTC m=+24.589147526" watchObservedRunningTime="2026-09-17 17:41:25.721707219 +0000 UTC m=+24.589840198"
Sep 17 17:41:26.305015 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.304901    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:25.411460386Z","UUID":"f015fdaf-2e10-43b4-a00c-7831e97c71a9","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:26.307112 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.307060    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:26.307112 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.307096    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:26.355894 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.355868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:26.356039 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:26.355980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:26.356110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.356041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:26.356175 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:26.356155    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:26.650731 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.650667    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:41:26.651137 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:26.651065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"8cedf13929db40a2a210a8ba1e24c62c8cf5d7d70424c60040462151059fac7d"}
Sep 17 17:41:27.356041 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:27.356012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:27.356241 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:27.356151    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:27.658329 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:27.658252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r" event={"ID":"6471513d-3e04-421b-8d90-7005d966cefc","Type":"ContainerStarted","Data":"46ccf0229844e02a883127ca8d9e101b0f961a5a04e5777bdf556d60539e8adc"}
Sep 17 17:41:27.680669 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:27.680615    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vk99r" podStartSLOduration=3.499158131 podStartE2EDuration="24.680596005s" totalImagesPullingTime="21.181437874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:27.67982531 +0000 UTC m=+26.547958273" watchObservedRunningTime="2026-09-17 17:41:27.680596005 +0000 UTC m=+26.548728967"
Sep 17 17:41:28.356139 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:28.356107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:28.356139 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:28.356148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:28.356355 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:28.356235    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:28.356431 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:28.356392    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:29.356158 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.356002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:29.356701 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:29.356243    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:29.666282 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.666215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:41:29.666591 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.666565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"3a55968c72f73e2f9edc22f6fbea60a62733a2fc4b2db70f84434344d1ae6a35"}
Sep 17 17:41:29.666908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.666880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:29.667029 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.666983    2581 scope.go:117] "RemoveContainer" containerID="7524ef3de512189c58f091a36dfdc34cc9c782bedfe90ebfda9efd071383be9a"
Sep 17 17:41:29.668680 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.668658    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="f4b72d88dfabc451cfb70b0d2c739d76c801d76adf019cd357abc2c6f1d2d48b" exitCode=0
Sep 17 17:41:29.668782 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.668696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"f4b72d88dfabc451cfb70b0d2c739d76c801d76adf019cd357abc2c6f1d2d48b"}
Sep 17 17:41:29.682698 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:29.682678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:30.355654 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.355615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:30.355806 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.355656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:30.355806 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:30.355769    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:30.355924 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:30.355904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:30.598512 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.598308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4cbwc"]
Sep 17 17:41:30.602670 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.602630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rspk"]
Sep 17 17:41:30.602961 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.602936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfzxt"]
Sep 17 17:41:30.603073 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.603058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:30.603186 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:30.603164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:30.674053 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:41:30.674383 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:30.674481 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:30.674481 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" event={"ID":"db7e86f5-234c-409d-96c1-a57c43db924e","Type":"ContainerStarted","Data":"7365f416ea6ee9e2de7af82e72a7857c006d5e9b0e60fabbc2b511e44fdb3d30"}
Sep 17 17:41:30.674583 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:30.674522    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:30.674583 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:30.674557    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:30.674712 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:30.674774 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.674760    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:30.689187 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.689160    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:41:30.736907 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:30.736865    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl" podStartSLOduration=8.963413896 podStartE2EDuration="27.736851951s" totalImagesPullingTime="18.773438055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.736536295 +0000 UTC m=+29.604669240" watchObservedRunningTime="2026-09-17 17:41:30.736851951 +0000 UTC m=+29.604984921"
Sep 17 17:41:31.678523 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:31.678484    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="df2f2137aee9825eabd7b52c3751da770d054ff328819afdef0f58b94741f5af" exitCode=0
Sep 17 17:41:31.678890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:31.678558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"df2f2137aee9825eabd7b52c3751da770d054ff328819afdef0f58b94741f5af"}
Sep 17 17:41:32.356320 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:32.356290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:32.356505 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:32.356397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:32.356505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:32.356290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:32.356505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:32.356290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:32.356505 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:32.356477    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:32.356676 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:32.356538    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:33.684432 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:33.684402    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="91585c02891f4272dfc379b5837f2f3ae45037c90f4fc8c8bb3b3215f8b3d3c4" exitCode=0
Sep 17 17:41:33.684858 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:33.684450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"91585c02891f4272dfc379b5837f2f3ae45037c90f4fc8c8bb3b3215f8b3d3c4"}
Sep 17 17:41:34.355631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:34.355594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:34.355631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:34.355635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:34.355829 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:34.355602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:34.355829 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:34.355740    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:34.355911 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:34.355822    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:34.355911 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:34.355884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:35.452354 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:35.452312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.551107 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:35.551068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.980524 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:35.980484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:35.980754 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:35.980532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:35.980754 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:35.980637    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:35.980754 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:35.980659    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:35.980754 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:35.980710    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs podName:8215753a-c993-4f2a-82d1-756ab30df519 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:07.980693832 +0000 UTC m=+66.848826774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs") pod "network-metrics-daemon-4cbwc" (UID: "8215753a-c993-4f2a-82d1-756ab30df519") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:35.980754 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:35.980723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret podName:1e889876-aac5-4dda-ad6d-cf30f00dc6ab nodeName:}" failed. No retries permitted until 2026-09-17 17:42:07.980717714 +0000 UTC m=+66.848850655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret") pod "global-pull-secret-syncer-5rspk" (UID: "1e889876-aac5-4dda-ad6d-cf30f00dc6ab") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:36.081839 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.081807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:36.082065 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.081987    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:36.082065 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.082010    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:36.082065 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.082023    2581 projected.go:196] Error preparing data for projected volume kube-api-access-ch8k7 for pod openshift-network-diagnostics/network-check-target-lfzxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:36.082169 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.082083    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7 podName:e8fa2840-f25a-4ab4-b208-4a54ae5dfafc nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.082066542 +0000 UTC m=+66.950199484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ch8k7" (UniqueName: "kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7") pod "network-check-target-lfzxt" (UID: "e8fa2840-f25a-4ab4-b208-4a54ae5dfafc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:36.356330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.356300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:36.356509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.356358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:36.356509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.356405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:36.356509 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.356428    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfzxt" podUID="e8fa2840-f25a-4ab4-b208-4a54ae5dfafc"
Sep 17 17:41:36.356691 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.356506    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cbwc" podUID="8215753a-c993-4f2a-82d1-756ab30df519"
Sep 17 17:41:36.356691 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.356584    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rspk" podUID="1e889876-aac5-4dda-ad6d-cf30f00dc6ab"
Sep 17 17:41:36.531278 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.531248    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeReady"
Sep 17 17:41:36.531655 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.531414    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:36.595149 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.595121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cp6jw"]
Sep 17 17:41:36.598048 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.598029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.600594 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.600400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tw6cx"]
Sep 17 17:41:36.601196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.601173    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:36.601301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.601180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kcr8x\""
Sep 17 17:41:36.601301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.601219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:36.601451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.601429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:36.602452 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.602432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.605316 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.605300    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:36.605421 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.605411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:36.605534 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.605520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fglck\""
Sep 17 17:41:36.611448 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.611430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cp6jw"]
Sep 17 17:41:36.617196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.617178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tw6cx"]
Sep 17 17:41:36.685254 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.685225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c9ad593-3c0f-4a10-aa7f-17972148edd9-cert\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.685394 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.685300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhqn\" (UniqueName: \"kubernetes.io/projected/2c9ad593-3c0f-4a10-aa7f-17972148edd9-kube-api-access-7hhqn\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.786134 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.786319 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587489eb-4e77-4bb2-b22a-8ed06399d34b-config-volume\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.786319 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhqn\" (UniqueName: \"kubernetes.io/projected/2c9ad593-3c0f-4a10-aa7f-17972148edd9-kube-api-access-7hhqn\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.786319 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c9ad593-3c0f-4a10-aa7f-17972148edd9-cert\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.786474 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587489eb-4e77-4bb2-b22a-8ed06399d34b-tmp-dir\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.786474 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.786373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhm8\" (UniqueName: \"kubernetes.io/projected/587489eb-4e77-4bb2-b22a-8ed06399d34b-kube-api-access-twhm8\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.790738 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.790713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c9ad593-3c0f-4a10-aa7f-17972148edd9-cert\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.796006 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.795980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhqn\" (UniqueName: \"kubernetes.io/projected/2c9ad593-3c0f-4a10-aa7f-17972148edd9-kube-api-access-7hhqn\") pod \"ingress-canary-cp6jw\" (UID: \"2c9ad593-3c0f-4a10-aa7f-17972148edd9\") " pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:36.887301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.887301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587489eb-4e77-4bb2-b22a-8ed06399d34b-config-volume\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.887503 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587489eb-4e77-4bb2-b22a-8ed06399d34b-tmp-dir\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.887503 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhm8\" (UniqueName: \"kubernetes.io/projected/587489eb-4e77-4bb2-b22a-8ed06399d34b-kube-api-access-twhm8\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.887503 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.887387    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.887503 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:36.887468    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls podName:587489eb-4e77-4bb2-b22a-8ed06399d34b nodeName:}" failed. No retries permitted until 2026-09-17 17:41:37.387447238 +0000 UTC m=+36.255580199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls") pod "dns-default-tw6cx" (UID: "587489eb-4e77-4bb2-b22a-8ed06399d34b") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.887862 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587489eb-4e77-4bb2-b22a-8ed06399d34b-tmp-dir\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.887974 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.887894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587489eb-4e77-4bb2-b22a-8ed06399d34b-config-volume\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.897589 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.897559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhm8\" (UniqueName: \"kubernetes.io/projected/587489eb-4e77-4bb2-b22a-8ed06399d34b-kube-api-access-twhm8\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:36.911534 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:36.911509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cp6jw"
Sep 17 17:41:37.144767 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:37.144674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:37.211655 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:37.211604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:37.392601 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:37.392569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:37.392805 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:37.392777    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:37.392885 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:37.392871    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls podName:587489eb-4e77-4bb2-b22a-8ed06399d34b nodeName:}" failed. No retries permitted until 2026-09-17 17:41:38.392850774 +0000 UTC m=+37.260983719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls") pod "dns-default-tw6cx" (UID: "587489eb-4e77-4bb2-b22a-8ed06399d34b") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:37.870626 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:37.870566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:37.965630 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:37.965599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:38.081346 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.081279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cp6jw"]
Sep 17 17:41:38.084202 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:38.084177    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9ad593_3c0f_4a10_aa7f_17972148edd9.slice/crio-7d371ba34e2babb7399eade21c534a318660a9ec6df273550d7b268e44e71e7d WatchSource:0}: Error finding container 7d371ba34e2babb7399eade21c534a318660a9ec6df273550d7b268e44e71e7d: Status 404 returned error can't find the container with id 7d371ba34e2babb7399eade21c534a318660a9ec6df273550d7b268e44e71e7d
Sep 17 17:41:38.355538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.355450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:41:38.355722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.355458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:41:38.355722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.355467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:41:38.360269 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360240    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:38.360269 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360252    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:38.360472 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gwk7t\""
Sep 17 17:41:38.360472 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360295    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:38.360472 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360254    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:38.360472 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.360253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hq5mj\""
Sep 17 17:41:38.399772 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.399744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:38.399903 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:38.399865    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:38.399939 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:38.399929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls podName:587489eb-4e77-4bb2-b22a-8ed06399d34b nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.399915138 +0000 UTC m=+39.268048093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls") pod "dns-default-tw6cx" (UID: "587489eb-4e77-4bb2-b22a-8ed06399d34b") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:38.700619 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.700529    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="beb380a6d29baa9cfa59e4f82f06720b362164b4f5512b79ac66b0afc55dbc89" exitCode=0
Sep 17 17:41:38.700782 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.700621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"beb380a6d29baa9cfa59e4f82f06720b362164b4f5512b79ac66b0afc55dbc89"}
Sep 17 17:41:38.702188 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:38.702150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cp6jw" event={"ID":"2c9ad593-3c0f-4a10-aa7f-17972148edd9","Type":"ContainerStarted","Data":"7d371ba34e2babb7399eade21c534a318660a9ec6df273550d7b268e44e71e7d"}
Sep 17 17:41:39.709130 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:39.709064    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a4a871b-b563-4ba1-b00e-d0687edf4d03" containerID="686e90baedb2c5a49235822753a92eac20a5757b93ab117cb50c19c7bc125f41" exitCode=0
Sep 17 17:41:39.709130 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:39.709102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerDied","Data":"686e90baedb2c5a49235822753a92eac20a5757b93ab117cb50c19c7bc125f41"}
Sep 17 17:41:40.414114 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.413670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:40.414114 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:40.413822    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.414114 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:40.414068    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls podName:587489eb-4e77-4bb2-b22a-8ed06399d34b nodeName:}" failed. No retries permitted until 2026-09-17 17:41:44.41404688 +0000 UTC m=+43.282179840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls") pod "dns-default-tw6cx" (UID: "587489eb-4e77-4bb2-b22a-8ed06399d34b") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.682524 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.682490    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.716104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.716071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" event={"ID":"2a4a871b-b563-4ba1-b00e-d0687edf4d03","Type":"ContainerStarted","Data":"3b2ce085e1fe18eca5475df3a89827ca125dc0c5f925513c87a66148ef9a820d"}
Sep 17 17:41:40.717605 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.717582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cp6jw" event={"ID":"2c9ad593-3c0f-4a10-aa7f-17972148edd9","Type":"ContainerStarted","Data":"8723ee2c51a140dc06105d8ee6cfbda2def2a95c4e397dd97aa11622e3738019"}
Sep 17 17:41:40.742708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.742657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nmzvx" podStartSLOduration=8.313137242 podStartE2EDuration="37.74262326s" totalImagesPullingTime="29.429486018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:40.740762479 +0000 UTC m=+39.608895441" watchObservedRunningTime="2026-09-17 17:41:40.74262326 +0000 UTC m=+39.610756291"
Sep 17 17:41:40.759141 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.759096    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cp6jw" podStartSLOduration=2.251443675 podStartE2EDuration="4.759081604s" totalImagesPullingTime="2.507637929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:40.75841908 +0000 UTC m=+39.626552043" watchObservedRunningTime="2026-09-17 17:41:40.759081604 +0000 UTC m=+39.627214567"
Sep 17 17:41:40.839769 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.839740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.942277 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:40.942192    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.041021 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:41.040990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.221860 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:41.221793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.289026 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:41.288982    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.500374 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:41.500296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.575465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:41.575432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.170892 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.170858    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzz5j"]
Sep 17 17:41:42.179024 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.179003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.181925 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.181898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:41:42.182068 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.181898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:41:42.182068 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.181947    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cmvbs\""
Sep 17 17:41:42.182068 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.181956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:41:42.183378 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.183358    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:41:42.185731 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.185710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzz5j"]
Sep 17 17:41:42.196445 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.196421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.273497 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.273463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.328189 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.328151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.328341 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.328196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.328341 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.328221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89d826d4-b800-4639-9e65-e6678e7fd081-crio-socket\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.328341 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.328300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4nv\" (UniqueName: \"kubernetes.io/projected/89d826d4-b800-4639-9e65-e6678e7fd081-kube-api-access-pr4nv\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.328480 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.328367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89d826d4-b800-4639-9e65-e6678e7fd081-data-volume\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.401192 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.401157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.429257 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429257 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89d826d4-b800-4639-9e65-e6678e7fd081-crio-socket\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4nv\" (UniqueName: \"kubernetes.io/projected/89d826d4-b800-4639-9e65-e6678e7fd081-kube-api-access-pr4nv\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.429308    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.429379    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.429410    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.929390103 +0000 UTC m=+41.797523066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.429426 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.429425    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.92941782 +0000 UTC m=+41.797550762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.429611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89d826d4-b800-4639-9e65-e6678e7fd081-data-volume\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89d826d4-b800-4639-9e65-e6678e7fd081-crio-socket\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.429845 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.429828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89d826d4-b800-4639-9e65-e6678e7fd081-data-volume\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.437736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.437704    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"]
Sep 17 17:41:42.448393 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.448362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4nv\" (UniqueName: \"kubernetes.io/projected/89d826d4-b800-4639-9e65-e6678e7fd081-kube-api-access-pr4nv\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.452560 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.452537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"]
Sep 17 17:41:42.452665 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.452632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"
Sep 17 17:41:42.455807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.455786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:41:42.455927 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.455786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ppp2v\""
Sep 17 17:41:42.455927 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.455792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:42.478430 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.478398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.534238 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.534204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.631411 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.631380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jzr\" (UniqueName: \"kubernetes.io/projected/77002440-0a3c-46bc-b7c9-d26de142d234-kube-api-access-h5jzr\") pod \"migrator-76794cbd6c-m84bj\" (UID: \"77002440-0a3c-46bc-b7c9-d26de142d234\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"
Sep 17 17:41:42.650105 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.650067    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:42.732660 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.732562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jzr\" (UniqueName: \"kubernetes.io/projected/77002440-0a3c-46bc-b7c9-d26de142d234-kube-api-access-h5jzr\") pod \"migrator-76794cbd6c-m84bj\" (UID: \"77002440-0a3c-46bc-b7c9-d26de142d234\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"
Sep 17 17:41:42.742827 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.742799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jzr\" (UniqueName: \"kubernetes.io/projected/77002440-0a3c-46bc-b7c9-d26de142d234-kube-api-access-h5jzr\") pod \"migrator-76794cbd6c-m84bj\" (UID: \"77002440-0a3c-46bc-b7c9-d26de142d234\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"
Sep 17 17:41:42.761701 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.761669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"
Sep 17 17:41:42.934077 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.934043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.934241 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:42.934087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:42.934241 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.934169    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.934241 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.934227    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.934241 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.934238    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:43.934221573 +0000 UTC m=+42.802354519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.934367 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:42.934271    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:43.934260148 +0000 UTC m=+42.802393108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:43.063376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.063347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj"]
Sep 17 17:41:43.066223 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:43.066184    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77002440_0a3c_46bc_b7c9_d26de142d234.slice/crio-65edd60a35afc1b9015d464a10f7cb1433b6eba121c44fc76c9d917d290b53df WatchSource:0}: Error finding container 65edd60a35afc1b9015d464a10f7cb1433b6eba121c44fc76c9d917d290b53df: Status 404 returned error can't find the container with id 65edd60a35afc1b9015d464a10f7cb1433b6eba121c44fc76c9d917d290b53df
Sep 17 17:41:43.154510 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.154469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.244054 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.244019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.323881 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.323792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.727048 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.726970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj" event={"ID":"77002440-0a3c-46bc-b7c9-d26de142d234","Type":"ContainerStarted","Data":"65edd60a35afc1b9015d464a10f7cb1433b6eba121c44fc76c9d917d290b53df"}
Sep 17 17:41:43.942042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.941997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:43.942237 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:43.942062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:43.942237 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:43.942134    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:43.942237 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:43.942224    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:45.942201258 +0000 UTC m=+44.810334202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:43.942237 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:43.942228    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:43.942459 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:43.942292    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:45.942273907 +0000 UTC m=+44.810406863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:44.133506 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:44.133473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhwgg_c3029b88-7089-4632-bd93-a6323afb9b49/dns-node-resolver/0.log"
Sep 17 17:41:44.446470 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:44.446386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:44.446866 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:44.446541    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.446866 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:44.446607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls podName:587489eb-4e77-4bb2-b22a-8ed06399d34b nodeName:}" failed. No retries permitted until 2026-09-17 17:41:52.44659007 +0000 UTC m=+51.314723011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls") pod "dns-default-tw6cx" (UID: "587489eb-4e77-4bb2-b22a-8ed06399d34b") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.940320 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:44.940280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.014084 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.013998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.132227 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.132201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gjbf_47daedcd-8541-40ae-847e-7b786c4a982b/node-ca/0.log"
Sep 17 17:41:45.287499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.287465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.357103 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.357065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.732924 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.732835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj" event={"ID":"77002440-0a3c-46bc-b7c9-d26de142d234","Type":"ContainerStarted","Data":"f4ef2225ed70724d99d31f6a7d9a835ae711a2b9b18532cbf2b5f3b191d6746d"}
Sep 17 17:41:45.732924 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.732895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj" event={"ID":"77002440-0a3c-46bc-b7c9-d26de142d234","Type":"ContainerStarted","Data":"a598faffab2dfe1659f2adb290f9a06fb778931d65fffa4f0d8bab7a10dab880"}
Sep 17 17:41:45.843676 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.843630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.931126 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.931066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cp6jw_2c9ad593-3c0f-4a10-aa7f-17972148edd9/serve-healthcheck-canary/0.log"
Sep 17 17:41:45.948346 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.948308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.957278 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.957248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:45.957396 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:45.957296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:45.957455 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:45.957388    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:45.957455 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:45.957425    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:45.957545 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:45.957456    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:49.957440559 +0000 UTC m=+48.825573525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:45.957545 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:45.957482    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:49.957465815 +0000 UTC m=+48.825598771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:46.908482 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:46.908444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.943752 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:47.943721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.609164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:48.609136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.710047 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:48.710020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.788787 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:48.788756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.868815 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:48.868737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.995567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:48.995533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.117895 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.117862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.415695 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.415659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.538444 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.538407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.612743 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.612714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.982664 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.982623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:49.982828 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.982676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:49.982828 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:49.982801    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:49.982910 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:41:49.982855    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls podName:89d826d4-b800-4639-9e65-e6678e7fd081 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.98284142 +0000 UTC m=+56.850974361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzz5j" (UID: "89d826d4-b800-4639-9e65-e6678e7fd081") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:49.983212 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:49.983193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89d826d4-b800-4639-9e65-e6678e7fd081-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:50.514814 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:50.514759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.712505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:50.712465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.908812 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:50.908773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.976420 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:50.976393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:51.109125 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:51.109092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:51.313096 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:51.313064    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:52.110797 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.110755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:52.316299 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.316260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:52.429885 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.429783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:52.499617 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.499584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:52.501815 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.501785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587489eb-4e77-4bb2-b22a-8ed06399d34b-metrics-tls\") pod \"dns-default-tw6cx\" (UID: \"587489eb-4e77-4bb2-b22a-8ed06399d34b\") " pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:52.517777 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.517751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:52.530953 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.530923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:52.815456 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.815397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-m84bj" podStartSLOduration=9.144617182 podStartE2EDuration="10.815379517s" totalImagesPullingTime="1.670762335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:45.753578972 +0000 UTC m=+44.621711935" watchObservedRunningTime="2026-09-17 17:41:52.815379517 +0000 UTC m=+51.683512480"
Sep 17 17:41:52.815732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.815716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tw6cx"]
Sep 17 17:41:52.819849 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:52.819820    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587489eb_4e77_4bb2_b22a_8ed06399d34b.slice/crio-acacced7253bafe8b869a403edb94252a7fca210a61040b69345e86519408133 WatchSource:0}: Error finding container acacced7253bafe8b869a403edb94252a7fca210a61040b69345e86519408133: Status 404 returned error can't find the container with id acacced7253bafe8b869a403edb94252a7fca210a61040b69345e86519408133
Sep 17 17:41:52.914198 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:52.914167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.324125 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.324089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.412680 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.412631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.477799 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.477761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.631261 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.631162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.733039 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.733003    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:53.753922 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.753874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tw6cx" event={"ID":"587489eb-4e77-4bb2-b22a-8ed06399d34b","Type":"ContainerStarted","Data":"acacced7253bafe8b869a403edb94252a7fca210a61040b69345e86519408133"}
Sep 17 17:41:53.913111 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:53.913039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.013487 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.013449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.320336 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.320304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.425719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.425684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.535317 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.535280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.633043 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.632981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.767623 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.767578    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.866105 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:54.866082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.120786 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.120757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.286099 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.286070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.519564 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.519532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.761956 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.761874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tw6cx" event={"ID":"587489eb-4e77-4bb2-b22a-8ed06399d34b","Type":"ContainerStarted","Data":"901ac0b1c124e67f61c1140f0830b5ddab5b83dff29a510a4ec608a2a134883f"}
Sep 17 17:41:55.761956 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.761920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tw6cx" event={"ID":"587489eb-4e77-4bb2-b22a-8ed06399d34b","Type":"ContainerStarted","Data":"ec22daef5c021657d23f3e837b5ec800304532107af538b5caa5cab972c33431"}
Sep 17 17:41:55.761956 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.761951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:41:55.790222 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.790179    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tw6cx" podStartSLOduration=17.886755954 podStartE2EDuration="19.790165423s" totalImagesPullingTime="1.903409469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:55.788524551 +0000 UTC m=+54.656657515" watchObservedRunningTime="2026-09-17 17:41:55.790165423 +0000 UTC m=+54.658298382"
Sep 17 17:41:55.958997 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:55.958967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.142312 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:56.142282    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.246090 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:56.246057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.355436 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:56.355406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:56.526773 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:56.526682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.313736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:57.313701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.001505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.001470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.036579 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.036541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:58.038876 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.038857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89d826d4-b800-4639-9e65-e6678e7fd081-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzz5j\" (UID: \"89d826d4-b800-4639-9e65-e6678e7fd081\") " pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:58.090475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.090446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzz5j"
Sep 17 17:41:58.157319 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.157288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.317155 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.317121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.359359 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.359334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzz5j"]
Sep 17 17:41:58.362268 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:41:58.362244    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d826d4_b800_4639_9e65_e6678e7fd081.slice/crio-ac459cda2a8d7bac1fbcb490c5d6876623ebcc026267a13b6b2ebcd767d3c763 WatchSource:0}: Error finding container ac459cda2a8d7bac1fbcb490c5d6876623ebcc026267a13b6b2ebcd767d3c763: Status 404 returned error can't find the container with id ac459cda2a8d7bac1fbcb490c5d6876623ebcc026267a13b6b2ebcd767d3c763
Sep 17 17:41:58.533519 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.533472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.636934 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.636853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.772788 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.772753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzz5j" event={"ID":"89d826d4-b800-4639-9e65-e6678e7fd081","Type":"ContainerStarted","Data":"bea289d93a5858177a2ac41fa82cb35f04686f69cd2c6663ffcb284e9e8a2b4b"}
Sep 17 17:41:58.772942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:58.772835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzz5j" event={"ID":"89d826d4-b800-4639-9e65-e6678e7fd081","Type":"ContainerStarted","Data":"ac459cda2a8d7bac1fbcb490c5d6876623ebcc026267a13b6b2ebcd767d3c763"}
Sep 17 17:41:59.311128 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:59.311093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.504595 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:59.504535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.710001 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:59.709923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.778303 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:41:59.778273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzz5j" event={"ID":"89d826d4-b800-4639-9e65-e6678e7fd081","Type":"ContainerStarted","Data":"40876b0addeb6f7d56542258c87b082f0dd5a6c6b025842fe1ffe268377fabdb"}
Sep 17 17:42:00.413540 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:00.413453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.517953 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:00.517921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.783458 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:00.783416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzz5j" event={"ID":"89d826d4-b800-4639-9e65-e6678e7fd081","Type":"ContainerStarted","Data":"537f12f3f6126cf8c48419278cc3fcf54ffe4ab2166fbc9638ab0a75db360d9b"}
Sep 17 17:42:00.815992 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:00.815950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bzz5j" podStartSLOduration=17.126596435 podStartE2EDuration="18.815935465s" totalImagesPullingTime="1.68933903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:00.815421658 +0000 UTC m=+59.683554621" watchObservedRunningTime="2026-09-17 17:42:00.815935465 +0000 UTC m=+59.684068421"
Sep 17 17:42:02.691363 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:02.691337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6ksl"
Sep 17 17:42:05.767481 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:05.767447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tw6cx"
Sep 17 17:42:07.033716 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.033681    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"]
Sep 17 17:42:07.036218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.036200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.039145 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.039124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4pqfq\""
Sep 17 17:42:07.039261 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.039127    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:42:07.039261 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.039127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:42:07.040740 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.040721    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:42:07.040831 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.040721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:42:07.041567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.041548    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"]
Sep 17 17:42:07.043353 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.043333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.047499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.047483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:42:07.059619 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.059593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"]
Sep 17 17:42:07.076622 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.076599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"]
Sep 17 17:42:07.103810 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.103790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/09397f91-6e68-484d-9fb7-d7cd5cdacfec-klusterlet-config\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.103908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.103817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5ec2dae-831c-43bb-988b-30ef5a99c1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.103908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.103835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmwp\" (UniqueName: \"kubernetes.io/projected/09397f91-6e68-484d-9fb7-d7cd5cdacfec-kube-api-access-xrmwp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.103908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.103857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v54fh\" (UniqueName: \"kubernetes.io/projected/e5ec2dae-831c-43bb-988b-30ef5a99c1af-kube-api-access-v54fh\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.103908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.103889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09397f91-6e68-484d-9fb7-d7cd5cdacfec-tmp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.164102 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.164074    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:07.166037 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.166022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.178538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:42:07.178666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:42:07.178730 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:42:07.178730 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:42:07.178730 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:42:07.178971 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kxlr\""
Sep 17 17:42:07.179020 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.178955    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:42:07.184194 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.184178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:42:07.189499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.189480    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-586df9fd45-769kt"]
Sep 17 17:42:07.191384 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.191370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.203885 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.203865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:07.204186 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.204228 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-tls\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204262 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-installation-pull-secrets\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204294 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.204327 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-trusted-ca\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204327 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5ec2dae-831c-43bb-988b-30ef5a99c1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.204389 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.204389 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.204389 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-bound-sa-token\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/09397f91-6e68-484d-9fb7-d7cd5cdacfec-klusterlet-config\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.204476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64179cb8-717e-4504-9007-45b11c1d9a2d-ca-trust-extracted\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204548 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmwp\" (UniqueName: \"kubernetes.io/projected/09397f91-6e68-484d-9fb7-d7cd5cdacfec-kube-api-access-xrmwp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.204548 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v54fh\" (UniqueName: \"kubernetes.io/projected/e5ec2dae-831c-43bb-988b-30ef5a99c1af-kube-api-access-v54fh\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.204669 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-certificates\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204737 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09397f91-6e68-484d-9fb7-d7cd5cdacfec-tmp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.204737 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-image-registry-private-configuration\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204838 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.204838 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfscz\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-kube-api-access-tfscz\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.204937 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.204864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.205206 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.205166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09397f91-6e68-484d-9fb7-d7cd5cdacfec-tmp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.206928 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.206908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/09397f91-6e68-484d-9fb7-d7cd5cdacfec-klusterlet-config\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.207056 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.207036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5ec2dae-831c-43bb-988b-30ef5a99c1af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.207564 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.207548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:42:07.207612 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.207574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:42:07.207889 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.207868    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:42:07.209339 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.209321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:42:07.218281 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.218261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-586df9fd45-769kt"]
Sep 17 17:42:07.218600 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.218580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mtf7c\""
Sep 17 17:42:07.250505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.250477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v54fh\" (UniqueName: \"kubernetes.io/projected/e5ec2dae-831c-43bb-988b-30ef5a99c1af-kube-api-access-v54fh\") pod \"managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2\" (UID: \"e5ec2dae-831c-43bb-988b-30ef5a99c1af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.253933 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.253913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmwp\" (UniqueName: \"kubernetes.io/projected/09397f91-6e68-484d-9fb7-d7cd5cdacfec-kube-api-access-xrmwp\") pod \"klusterlet-addon-workmgr-667d6dd7cd-hpq66\" (UID: \"09397f91-6e68-484d-9fb7-d7cd5cdacfec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.300331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.300263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.305196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-trusted-ca\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305297 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.305297 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.305297 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-bound-sa-token\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305408 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64179cb8-717e-4504-9007-45b11c1d9a2d-ca-trust-extracted\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305443 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-certificates\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305487 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-image-registry-private-configuration\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305610 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.305722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfscz\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-kube-api-access-tfscz\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.305722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.305879 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-tls\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305879 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-installation-pull-secrets\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.305879 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.305832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.306068 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.306221 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.306325 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-certificates\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.306531 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64179cb8-717e-4504-9007-45b11c1d9a2d-trusted-ca\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.306612 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.306691 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.306666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64179cb8-717e-4504-9007-45b11c1d9a2d-ca-trust-extracted\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.308251 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.308222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.308441 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.308424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.308568 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.308547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-image-registry-private-configuration\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.308630 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.308578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64179cb8-717e-4504-9007-45b11c1d9a2d-installation-pull-secrets\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.308727 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.308712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-registry-tls\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.324986 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.324966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj\") pod \"console-699d79656b-87v2t\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") " pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.342941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.342919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-bound-sa-token\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.348518 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.348489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfscz\" (UniqueName: \"kubernetes.io/projected/64179cb8-717e-4504-9007-45b11c1d9a2d-kube-api-access-tfscz\") pod \"image-registry-586df9fd45-769kt\" (UID: \"64179cb8-717e-4504-9007-45b11c1d9a2d\") " pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.356446 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.356429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"
Sep 17 17:42:07.363392 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.363373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:07.449471 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.449442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.474631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.474606    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:07.499317 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.499295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:07.681837 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.681578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2"]
Sep 17 17:42:07.682352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.682317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"]
Sep 17 17:42:07.684256 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:07.684230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ec2dae_831c_43bb_988b_30ef5a99c1af.slice/crio-da21050c77fc44cc82c6d0f9355f011250d9ee4ced067c3df3b2c950e3364d46 WatchSource:0}: Error finding container da21050c77fc44cc82c6d0f9355f011250d9ee4ced067c3df3b2c950e3364d46: Status 404 returned error can't find the container with id da21050c77fc44cc82c6d0f9355f011250d9ee4ced067c3df3b2c950e3364d46
Sep 17 17:42:07.684852 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:07.684832    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09397f91_6e68_484d_9fb7_d7cd5cdacfec.slice/crio-d86d1dfb917a9a26a751abea8844979982d5b300d013bb605dfb58f1aa57be1e WatchSource:0}: Error finding container d86d1dfb917a9a26a751abea8844979982d5b300d013bb605dfb58f1aa57be1e: Status 404 returned error can't find the container with id d86d1dfb917a9a26a751abea8844979982d5b300d013bb605dfb58f1aa57be1e
Sep 17 17:42:07.758495 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.758467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:07.761254 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:07.761230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddf28cf_52f6_4e83_b60f_d19f22386920.slice/crio-89767dc04891f596d5b81a58c51ca23064b93ff90131f53984a44c29dff66c2e WatchSource:0}: Error finding container 89767dc04891f596d5b81a58c51ca23064b93ff90131f53984a44c29dff66c2e: Status 404 returned error can't find the container with id 89767dc04891f596d5b81a58c51ca23064b93ff90131f53984a44c29dff66c2e
Sep 17 17:42:07.804317 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.804251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699d79656b-87v2t" event={"ID":"eddf28cf-52f6-4e83-b60f-d19f22386920","Type":"ContainerStarted","Data":"89767dc04891f596d5b81a58c51ca23064b93ff90131f53984a44c29dff66c2e"}
Sep 17 17:42:07.805380 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.805358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2" event={"ID":"e5ec2dae-831c-43bb-988b-30ef5a99c1af","Type":"ContainerStarted","Data":"da21050c77fc44cc82c6d0f9355f011250d9ee4ced067c3df3b2c950e3364d46"}
Sep 17 17:42:07.806404 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.806383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66" event={"ID":"09397f91-6e68-484d-9fb7-d7cd5cdacfec","Type":"ContainerStarted","Data":"d86d1dfb917a9a26a751abea8844979982d5b300d013bb605dfb58f1aa57be1e"}
Sep 17 17:42:07.820003 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:07.819981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-586df9fd45-769kt"]
Sep 17 17:42:07.823526 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:07.823507    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64179cb8_717e_4504_9007_45b11c1d9a2d.slice/crio-a5bbce1b49d0c53834ad5448fbeea7e082da573cbb3e45efc95c5a32813f9e7f WatchSource:0}: Error finding container a5bbce1b49d0c53834ad5448fbeea7e082da573cbb3e45efc95c5a32813f9e7f: Status 404 returned error can't find the container with id a5bbce1b49d0c53834ad5448fbeea7e082da573cbb3e45efc95c5a32813f9e7f
Sep 17 17:42:08.013664 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.013615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:42:08.013964 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.013696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:42:08.018532 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.018508    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:42:08.020814 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.020797    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:42:08.026969 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.026943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e889876-aac5-4dda-ad6d-cf30f00dc6ab-original-pull-secret\") pod \"global-pull-secret-syncer-5rspk\" (UID: \"1e889876-aac5-4dda-ad6d-cf30f00dc6ab\") " pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:42:08.027544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.027519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8215753a-c993-4f2a-82d1-756ab30df519-metrics-certs\") pod \"network-metrics-daemon-4cbwc\" (UID: \"8215753a-c993-4f2a-82d1-756ab30df519\") " pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:42:08.066061 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.065981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rspk"
Sep 17 17:42:08.076293 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.076264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.105627 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.105403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gwk7t\""
Sep 17 17:42:08.114233 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.114201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:42:08.115002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.114960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cbwc"
Sep 17 17:42:08.121547 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.121524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:42:08.135253 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.134917    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:42:08.137993 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.137971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8k7\" (UniqueName: \"kubernetes.io/projected/e8fa2840-f25a-4ab4-b208-4a54ae5dfafc-kube-api-access-ch8k7\") pod \"network-check-target-lfzxt\" (UID: \"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc\") " pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:42:08.254514 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.254466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.342715 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.342501    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rspk"]
Sep 17 17:42:08.346016 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:08.345985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e889876_aac5_4dda_ad6d_cf30f00dc6ab.slice/crio-8557e5c27e63eed8e99c010f9eb9ed7aef7a10ffa8a9c46bfb07e3a1c78f0414 WatchSource:0}: Error finding container 8557e5c27e63eed8e99c010f9eb9ed7aef7a10ffa8a9c46bfb07e3a1c78f0414: Status 404 returned error can't find the container with id 8557e5c27e63eed8e99c010f9eb9ed7aef7a10ffa8a9c46bfb07e3a1c78f0414
Sep 17 17:42:08.374048 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.374023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hq5mj\""
Sep 17 17:42:08.381793 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.381765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:42:08.414047 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.413962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4cbwc"]
Sep 17 17:42:08.681988 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.681917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfzxt"]
Sep 17 17:42:08.686102 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:08.685696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fa2840_f25a_4ab4_b208_4a54ae5dfafc.slice/crio-98691141e01041d9bb390bce3cbd814f3d9b945a5fa1175508d38673594e343b WatchSource:0}: Error finding container 98691141e01041d9bb390bce3cbd814f3d9b945a5fa1175508d38673594e343b: Status 404 returned error can't find the container with id 98691141e01041d9bb390bce3cbd814f3d9b945a5fa1175508d38673594e343b
Sep 17 17:42:08.821911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.821841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfzxt" event={"ID":"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc","Type":"ContainerStarted","Data":"98691141e01041d9bb390bce3cbd814f3d9b945a5fa1175508d38673594e343b"}
Sep 17 17:42:08.824628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.824572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cbwc" event={"ID":"8215753a-c993-4f2a-82d1-756ab30df519","Type":"ContainerStarted","Data":"ae15c6e65d7beda77d0d9b597b4ccaefe65d0690ee8545e21eb56e7163466c89"}
Sep 17 17:42:08.827921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.827785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-586df9fd45-769kt" event={"ID":"64179cb8-717e-4504-9007-45b11c1d9a2d","Type":"ContainerStarted","Data":"b1b03e47395deaf92d702343d59a7fafdbf0c63b627fed3347a5da9a1e72b18b"}
Sep 17 17:42:08.827921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.827838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-586df9fd45-769kt" event={"ID":"64179cb8-717e-4504-9007-45b11c1d9a2d","Type":"ContainerStarted","Data":"a5bbce1b49d0c53834ad5448fbeea7e082da573cbb3e45efc95c5a32813f9e7f"}
Sep 17 17:42:08.827921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.827897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:08.833414 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.833388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rspk" event={"ID":"1e889876-aac5-4dda-ad6d-cf30f00dc6ab","Type":"ContainerStarted","Data":"8557e5c27e63eed8e99c010f9eb9ed7aef7a10ffa8a9c46bfb07e3a1c78f0414"}
Sep 17 17:42:08.852776 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:08.851514    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-586df9fd45-769kt" podStartSLOduration=1.851494356 podStartE2EDuration="1.851494356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:08.851177836 +0000 UTC m=+67.719310799" watchObservedRunningTime="2026-09-17 17:42:08.851494356 +0000 UTC m=+67.719627313"
Sep 17 17:42:09.681740 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:09.681560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.816781 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:09.816656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.951482 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:09.951402    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.154451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:10.154417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.253266 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:10.253008    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:11.468597 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:11.468516    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:11.543904 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:11.543826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:12.778391 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:12.778304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:12.856706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:12.856671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:13.065528 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:13.065434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:13.141366 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:13.141334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:13.342700 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:13.342571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:13.437536 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:13.437496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.054896 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:16.054854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.160979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:16.160942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.255191 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:16.255105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:16.973096 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:16.973060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:17.092580 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:17.092542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:17.220256 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:17.220209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:19.328619 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.328580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:19.881687 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.881620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2" event={"ID":"e5ec2dae-831c-43bb-988b-30ef5a99c1af","Type":"ContainerStarted","Data":"1f1018121ff1f793e90bbd1ffb27052dd7d383bfda4c534dc2759eda6c4a74e1"}
Sep 17 17:42:19.883166 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.883134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rspk" event={"ID":"1e889876-aac5-4dda-ad6d-cf30f00dc6ab","Type":"ContainerStarted","Data":"51bdc65366ea1583d22554dbc0db219f24aff3367a63f7aeb1c8c65e4c2a55cb"}
Sep 17 17:42:19.884580 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.884554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66" event={"ID":"09397f91-6e68-484d-9fb7-d7cd5cdacfec","Type":"ContainerStarted","Data":"8ab859e19a8e284a177e1835dc7009b7e891fe2e57a206317c70fe8f0cfecd76"}
Sep 17 17:42:19.884706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.884686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:19.886415 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.886292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfzxt" event={"ID":"e8fa2840-f25a-4ab4-b208-4a54ae5dfafc","Type":"ContainerStarted","Data":"7c21e492395420beea2927bcdc0112c5784d3512dadcb8a19615ee5b04845de7"}
Sep 17 17:42:19.886415 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.886355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:42:19.886578 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.886541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66"
Sep 17 17:42:19.888034 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.888014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cbwc" event={"ID":"8215753a-c993-4f2a-82d1-756ab30df519","Type":"ContainerStarted","Data":"a5282395ff013103a6a9cd5b2954909a1a0282e2b7d624783b93eea0cc6ad58a"}
Sep 17 17:42:19.888121 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.888053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cbwc" event={"ID":"8215753a-c993-4f2a-82d1-756ab30df519","Type":"ContainerStarted","Data":"02f8754e7f360abb61b1acbaa2e52be4708440aafddc982b3428f680ae69dce3"}
Sep 17 17:42:19.889557 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.889531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699d79656b-87v2t" event={"ID":"eddf28cf-52f6-4e83-b60f-d19f22386920","Type":"ContainerStarted","Data":"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"}
Sep 17 17:42:19.905892 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.905851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c66c7dd4-r9hd2" podStartSLOduration=2.758807119 podStartE2EDuration="12.905838472s" totalImagesPullingTime="10.147031353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:19.905302633 +0000 UTC m=+78.773435596" watchObservedRunningTime="2026-09-17 17:42:19.905838472 +0000 UTC m=+78.773971434"
Sep 17 17:42:19.931095 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.931057    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5rspk" podStartSLOduration=66.406823885 podStartE2EDuration="1m16.931045307s" totalImagesPullingTime="10.524221422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:19.930905831 +0000 UTC m=+78.799038794" watchObservedRunningTime="2026-09-17 17:42:19.931045307 +0000 UTC m=+78.799178267"
Sep 17 17:42:19.956453 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.956414    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4cbwc" podStartSLOduration=68.215123411 podStartE2EDuration="1m16.956404775s" totalImagesPullingTime="8.741281364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:19.955990089 +0000 UTC m=+78.824123052" watchObservedRunningTime="2026-09-17 17:42:19.956404775 +0000 UTC m=+78.824537783"
Sep 17 17:42:19.992279 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:19.992237    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-699d79656b-87v2t" podStartSLOduration=1.8231731249999998 podStartE2EDuration="12.992224354s" totalImagesPullingTime="11.169051229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:19.991741772 +0000 UTC m=+78.859874734" watchObservedRunningTime="2026-09-17 17:42:19.992224354 +0000 UTC m=+78.860357316"
Sep 17 17:42:20.016515 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.016467    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lfzxt" podStartSLOduration=66.827154022 podStartE2EDuration="1m17.016453824s" totalImagesPullingTime="10.189299802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:20.015923361 +0000 UTC m=+78.884056326" watchObservedRunningTime="2026-09-17 17:42:20.016453824 +0000 UTC m=+78.884586787"
Sep 17 17:42:20.044299 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.044260    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-667d6dd7cd-hpq66" podStartSLOduration=1.8185433130000002 podStartE2EDuration="13.044246661s" totalImagesPullingTime="11.225703348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:20.042506714 +0000 UTC m=+78.910639678" watchObservedRunningTime="2026-09-17 17:42:20.044246661 +0000 UTC m=+78.912379660"
Sep 17 17:42:20.123719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.123688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.240519 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.240429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.614584 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.614554    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:20.775794 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.775749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.874297 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.874211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:20.942979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:20.942947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.171335 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.171256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.272837 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.272804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.515234 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.515125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.574790 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.574751    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4hjj5"]
Sep 17 17:42:21.577070 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.577054    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"]
Sep 17 17:42:21.577182 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.577167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.579256 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.579241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.580338 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.580320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:21.580401 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.580380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ql85w\""
Sep 17 17:42:21.580570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.580557    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:21.581425 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:42:21.581916 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:42:21.582002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581960    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:42:21.582002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581970    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:21.582002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581983    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:21.582148 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.581984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:42:21.582786 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.582772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mm6dm\""
Sep 17 17:42:21.583090 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.583076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:42:21.595081 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.595062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"]
Sep 17 17:42:21.624909 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.624877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-root\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.624921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.624971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-metrics-client-ca\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-wtmp\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-sys\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-textfile\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0565db71-00dd-4406-9807-f6be68ad05e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-tls\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4s7\" (UniqueName: \"kubernetes.io/projected/0484b5ea-a518-4abf-b94a-22837c9d2753-kube-api-access-5k4s7\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.625785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.625392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wmb\" (UniqueName: \"kubernetes.io/projected/0565db71-00dd-4406-9807-f6be68ad05e5-kube-api-access-h4wmb\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.648344 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.648296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.726274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-sys\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-textfile\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-sys\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0565db71-00dd-4406-9807-f6be68ad05e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-tls\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4s7\" (UniqueName: \"kubernetes.io/projected/0484b5ea-a518-4abf-b94a-22837c9d2753-kube-api-access-5k4s7\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wmb\" (UniqueName: \"kubernetes.io/projected/0565db71-00dd-4406-9807-f6be68ad05e5-kube-api-access-h4wmb\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-root\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-textfile\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-root\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-metrics-client-ca\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-wtmp\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.726921 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:42:21.726920    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.726964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0565db71-00dd-4406-9807-f6be68ad05e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:42:21.726991    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls podName:0565db71-00dd-4406-9807-f6be68ad05e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:22.226971857 +0000 UTC m=+81.095104801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-4bzn2" (UID: "0565db71-00dd-4406-9807-f6be68ad05e5") : secret "kube-state-metrics-tls" not found
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.727282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-wtmp\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.727294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.727287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.727352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.727294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0484b5ea-a518-4abf-b94a-22837c9d2753-metrics-client-ca\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.727681 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.727617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.729153 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.729127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.729444 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.729426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0484b5ea-a518-4abf-b94a-22837c9d2753-node-exporter-tls\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.729486 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.729449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.738854 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.738829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wmb\" (UniqueName: \"kubernetes.io/projected/0565db71-00dd-4406-9807-f6be68ad05e5-kube-api-access-h4wmb\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:21.744607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.744581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4s7\" (UniqueName: \"kubernetes.io/projected/0484b5ea-a518-4abf-b94a-22837c9d2753-kube-api-access-5k4s7\") pod \"node-exporter-4hjj5\" (UID: \"0484b5ea-a518-4abf-b94a-22837c9d2753\") " pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.767360 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.767301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:21.888763 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.888727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4hjj5"
Sep 17 17:42:21.899157 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:21.899125    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0484b5ea_a518_4abf_b94a_22837c9d2753.slice/crio-763897ade21b73e1b24a98fe311ffbb315c1041a449bcdfeeb35f4a6a8bf6aa4 WatchSource:0}: Error finding container 763897ade21b73e1b24a98fe311ffbb315c1041a449bcdfeeb35f4a6a8bf6aa4: Status 404 returned error can't find the container with id 763897ade21b73e1b24a98fe311ffbb315c1041a449bcdfeeb35f4a6a8bf6aa4
Sep 17 17:42:21.967101 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:21.967061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.137966 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.137928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.229911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.229877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.231847 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.231823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:22.234249 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.234230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0565db71-00dd-4406-9807-f6be68ad05e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4bzn2\" (UID: \"0565db71-00dd-4406-9807-f6be68ad05e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:22.329383 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.329345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.444142 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.444053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.492676 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.492625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"
Sep 17 17:42:22.551764 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.551728    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:22.554470 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.554446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.554935 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.554908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.558188 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:42:22.558289 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558272    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:42:22.558584 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558563    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:42:22.558695 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:42:22.558895 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558875    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:42:22.558965 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558950    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:42:22.559012 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:42:22.559012 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.558979    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nps7c\""
Sep 17 17:42:22.559265 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.559244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:42:22.559354 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.559303    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:42:22.575544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.575523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:22.636331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-web-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-config-out\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7twn\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-kube-api-access-k7twn\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.636785 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.636740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.644883 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.644848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:22.737236 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-web-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-config-out\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7twn\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-kube-api-access-k7twn\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737373 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737776 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737895 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.737971 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.737952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.738023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.738006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.738089 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.738053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.738311 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.738286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.738396 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.738372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.739476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.739376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/203923fa-ab2e-4747-852c-f89be5609ddc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.740079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.740053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/203923fa-ab2e-4747-852c-f89be5609ddc-config-out\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.741214 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.740686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.741214 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.741061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-config-volume\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.741486 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.741465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-web-config\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.741486 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.741481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.742041 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.742012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.742445 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.742424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.742490 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.742449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.742622 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.742606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/203923fa-ab2e-4747-852c-f89be5609ddc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.747960 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.747939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7twn\" (UniqueName: \"kubernetes.io/projected/203923fa-ab2e-4747-852c-f89be5609ddc-kube-api-access-k7twn\") pod \"alertmanager-main-0\" (UID: \"203923fa-ab2e-4747-852c-f89be5609ddc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.865621 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.865597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:42:22.902778 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.902747    2581 generic.go:356] "Generic (PLEG): container finished" podID="0484b5ea-a518-4abf-b94a-22837c9d2753" containerID="9a1a8d09a2581611929a2b867592b23158bb4999f2da3e2865683b4143aae5ed" exitCode=0
Sep 17 17:42:22.902951 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.902785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hjj5" event={"ID":"0484b5ea-a518-4abf-b94a-22837c9d2753","Type":"ContainerDied","Data":"9a1a8d09a2581611929a2b867592b23158bb4999f2da3e2865683b4143aae5ed"}
Sep 17 17:42:22.902951 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.902854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hjj5" event={"ID":"0484b5ea-a518-4abf-b94a-22837c9d2753","Type":"ContainerStarted","Data":"763897ade21b73e1b24a98fe311ffbb315c1041a449bcdfeeb35f4a6a8bf6aa4"}
Sep 17 17:42:22.954820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:22.954794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4bzn2"]
Sep 17 17:42:22.957958 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:22.957932    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0565db71_00dd_4406_9807_f6be68ad05e5.slice/crio-18da2b47909c16da1b2852e16e93582e9963e393ab4b30b063319ace0d34964e WatchSource:0}: Error finding container 18da2b47909c16da1b2852e16e93582e9963e393ab4b30b063319ace0d34964e: Status 404 returned error can't find the container with id 18da2b47909c16da1b2852e16e93582e9963e393ab4b30b063319ace0d34964e
Sep 17 17:42:23.158795 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.158768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:42:23.161160 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:23.161133    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203923fa_ab2e_4747_852c_f89be5609ddc.slice/crio-6f49d8d72daea18a3e94791a679d6a0ea1e5e39b26a4dfc1a04007193a62e2bf WatchSource:0}: Error finding container 6f49d8d72daea18a3e94791a679d6a0ea1e5e39b26a4dfc1a04007193a62e2bf: Status 404 returned error can't find the container with id 6f49d8d72daea18a3e94791a679d6a0ea1e5e39b26a4dfc1a04007193a62e2bf
Sep 17 17:42:23.908098 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.908061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"6f49d8d72daea18a3e94791a679d6a0ea1e5e39b26a4dfc1a04007193a62e2bf"}
Sep 17 17:42:23.909687 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.909620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2" event={"ID":"0565db71-00dd-4406-9807-f6be68ad05e5","Type":"ContainerStarted","Data":"18da2b47909c16da1b2852e16e93582e9963e393ab4b30b063319ace0d34964e"}
Sep 17 17:42:23.911979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.911954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hjj5" event={"ID":"0484b5ea-a518-4abf-b94a-22837c9d2753","Type":"ContainerStarted","Data":"c581952ccb94a5e66640f2d35d11c623bbe991edb3825ca198a8522f1b817d42"}
Sep 17 17:42:23.912099 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.912008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hjj5" event={"ID":"0484b5ea-a518-4abf-b94a-22837c9d2753","Type":"ContainerStarted","Data":"81c236aae6f82d0fa8670542b2649e559dc03260432245cba691855d60f7693f"}
Sep 17 17:42:23.944080 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:23.944036    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4hjj5" podStartSLOduration=2.092354163 podStartE2EDuration="2.944022863s" totalImagesPullingTime="792.158198ms" totalInitContainerRuntime="59.510502ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:23.942717971 +0000 UTC m=+82.810850935" watchObservedRunningTime="2026-09-17 17:42:23.944022863 +0000 UTC m=+82.812155826"
Sep 17 17:42:24.414337 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.414307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:24.515669 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.515581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:24.630559 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.630518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:24.730960 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.730924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:24.917455 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.917426    2581 generic.go:356] "Generic (PLEG): container finished" podID="203923fa-ab2e-4747-852c-f89be5609ddc" containerID="4b87930f4a45097fc02e29de69d19b12408735a02bc7162a216ad78eacfe59d3" exitCode=0
Sep 17 17:42:24.917878 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.917518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerDied","Data":"4b87930f4a45097fc02e29de69d19b12408735a02bc7162a216ad78eacfe59d3"}
Sep 17 17:42:24.919786 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.919760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2" event={"ID":"0565db71-00dd-4406-9807-f6be68ad05e5","Type":"ContainerStarted","Data":"e70a6b405284fc9063803a49721f1782a35376772ce598fa4a70ebadacff3033"}
Sep 17 17:42:24.919884 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.919867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2" event={"ID":"0565db71-00dd-4406-9807-f6be68ad05e5","Type":"ContainerStarted","Data":"9c8486d43e8315dd84c908fab5bffeabedbeef9a02e6dfa22418653bdfd61529"}
Sep 17 17:42:24.919932 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:24.919914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2" event={"ID":"0565db71-00dd-4406-9807-f6be68ad05e5","Type":"ContainerStarted","Data":"ae93f2d63b6f0bb00abdeea6b89a489c80fdb3a1a48cfde73418d766533aaf9d"}
Sep 17 17:42:26.217566 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.217537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.268453 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.268386    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-4bzn2" podStartSLOduration=4.010009219 podStartE2EDuration="5.268363556s" totalImagesPullingTime="1.258354337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:25.035625254 +0000 UTC m=+83.903758220" watchObservedRunningTime="2026-09-17 17:42:26.268363556 +0000 UTC m=+85.136496519"
Sep 17 17:42:26.269754 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.269734    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64c4d86f84-ssf8j"]
Sep 17 17:42:26.272683 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.272666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.277502 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.277485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-btukat2fs16o\""
Sep 17 17:42:26.277598 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.277569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ql8sk\""
Sep 17 17:42:26.277680 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.277593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:42:26.277920 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.277897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:42:26.278002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.277988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:42:26.278106 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.278089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:42:26.286988 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.286969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64c4d86f84-ssf8j"]
Sep 17 17:42:26.320433 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.320406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.369741 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvrt\" (UniqueName: \"kubernetes.io/projected/7f19eedf-1564-42a6-b7df-17521aa1a277-kube-api-access-tbvrt\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369741 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-tls\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369741 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7f19eedf-1564-42a6-b7df-17521aa1a277-audit-log\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-client-certs\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-metrics-server-audit-profiles\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.369947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.369857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-client-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.419352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.419324    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.470223 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-client-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470223 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvrt\" (UniqueName: \"kubernetes.io/projected/7f19eedf-1564-42a6-b7df-17521aa1a277-kube-api-access-tbvrt\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-tls\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7f19eedf-1564-42a6-b7df-17521aa1a277-audit-log\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470481 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-client-certs\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470577 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-metrics-server-audit-profiles\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.470610 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.470572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7f19eedf-1564-42a6-b7df-17521aa1a277-audit-log\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.471295 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.471270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.471505 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.471482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7f19eedf-1564-42a6-b7df-17521aa1a277-metrics-server-audit-profiles\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.472701 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.472678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-tls\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.473042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.473021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-secret-metrics-server-client-certs\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.473524 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.473504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7f19eedf-1564-42a6-b7df-17521aa1a277-client-ca-bundle\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.482509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.482491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvrt\" (UniqueName: \"kubernetes.io/projected/7f19eedf-1564-42a6-b7df-17521aa1a277-kube-api-access-tbvrt\") pod \"metrics-server-64c4d86f84-ssf8j\" (UID: \"7f19eedf-1564-42a6-b7df-17521aa1a277\") " pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.581821 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.581796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:26.726396 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.726318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.829884 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.829846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.856851 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.856827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64c4d86f84-ssf8j"]
Sep 17 17:42:26.858913 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:42:26.858888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f19eedf_1564_42a6_b7df_17521aa1a277.slice/crio-2d41c25303a044b1945511e235c351cb55d3755d2a0d59ebad112caf86bddfbc WatchSource:0}: Error finding container 2d41c25303a044b1945511e235c351cb55d3755d2a0d59ebad112caf86bddfbc: Status 404 returned error can't find the container with id 2d41c25303a044b1945511e235c351cb55d3755d2a0d59ebad112caf86bddfbc
Sep 17 17:42:26.931689 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.931632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"22eeeabfe74a2aa33ade6937e1b3e6a7e22c0d01d396223f52176729155ba6a2"}
Sep 17 17:42:26.931823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.931740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"b971a0f8f9a15ff6c3b24fe2de6a3f0877a2dde60ec29b6c4ebe8826b39bbdb5"}
Sep 17 17:42:26.931823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.931780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"b34f829482ec55dec88210d8dcc95fc777f7e0cf370809359f7e229a220df2af"}
Sep 17 17:42:26.931823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.931822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"2b3389add070bc8f8fe04614aff086b8f2577969b5ab6a87bb19c4d2b8ab21d6"}
Sep 17 17:42:26.932001 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.931862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"48dffa1b3026984fbf8190acb0232b682b052624d9696aa326d76cf3f691b797"}
Sep 17 17:42:26.933089 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:26.933061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j" event={"ID":"7f19eedf-1564-42a6-b7df-17521aa1a277","Type":"ContainerStarted","Data":"2d41c25303a044b1945511e235c351cb55d3755d2a0d59ebad112caf86bddfbc"}
Sep 17 17:42:27.324936 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.324892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.428727 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.428623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.475398 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.475369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:27.532086 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.532043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.812672 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.812621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.942466 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.942432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"203923fa-ab2e-4747-852c-f89be5609ddc","Type":"ContainerStarted","Data":"0be511c72ee28286baef4fe2fcd0c53b3680b6bc418f29f826c43cf7e21706db"}
Sep 17 17:42:27.981782 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:27.981733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9410985099999998 podStartE2EDuration="5.981713747s" totalImagesPullingTime="3.015590044s" totalInitContainerRuntime="25.025193ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:27.978808094 +0000 UTC m=+86.846941057" watchObservedRunningTime="2026-09-17 17:42:27.981713747 +0000 UTC m=+86.849846710"
Sep 17 17:42:28.949554 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:28.949520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j" event={"ID":"7f19eedf-1564-42a6-b7df-17521aa1a277","Type":"ContainerStarted","Data":"211b098fd52778d3f437f7a9cb931ce7aff8b12006a400d2fd550f596ee4abd1"}
Sep 17 17:42:28.970836 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:28.970790    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j" podStartSLOduration=1.7489320240000001 podStartE2EDuration="2.970772548s" totalImagesPullingTime="1.221840524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:28.9698003 +0000 UTC m=+87.837933264" watchObservedRunningTime="2026-09-17 17:42:28.970772548 +0000 UTC m=+87.838905511"
Sep 17 17:42:29.695432 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:29.695400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:29.767981 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:29.767942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:30.852384 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:30.852352    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-586df9fd45-769kt"
Sep 17 17:42:31.054801 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:31.054769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:31.181485 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:31.181415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:36.138176 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:36.138136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:36.244136 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:36.244101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.168761 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:37.168730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.267023 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:37.266989    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.327384 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:37.327350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:45.296157 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:45.296119    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:45.370562 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:45.370527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:46.582203 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:46.582174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:46.582732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:46.582244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:42:46.915588 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:46.915481    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-699d79656b-87v2t" podUID="eddf28cf-52f6-4e83-b60f-d19f22386920" containerName="console" containerID="cri-o://f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04" gracePeriod=15
Sep 17 17:42:47.182685 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.182596    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:47.335196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335372 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335372 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335372 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335299    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335372 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335600 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj\") pod \"eddf28cf-52f6-4e83-b60f-d19f22386920\" (UID: \"eddf28cf-52f6-4e83-b60f-d19f22386920\") "
Sep 17 17:42:47.335708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config" (OuterVolumeSpecName: "console-config") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:47.335708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca" (OuterVolumeSpecName: "service-ca") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:47.335837 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.335752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:47.337605 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.337579    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:47.337751 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.337623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj" (OuterVolumeSpecName: "kube-api-access-4gccj") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "kube-api-access-4gccj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:47.337751 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.337662    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eddf28cf-52f6-4e83-b60f-d19f22386920" (UID: "eddf28cf-52f6-4e83-b60f-d19f22386920"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:47.436367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436305    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/eddf28cf-52f6-4e83-b60f-d19f22386920-kube-api-access-4gccj\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:47.436367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436329    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-oauth-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:47.436367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436342    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-service-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:47.436367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436354    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddf28cf-52f6-4e83-b60f-d19f22386920-console-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:47.436367 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436366    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-console-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:47.436599 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:47.436379    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddf28cf-52f6-4e83-b60f-d19f22386920-oauth-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:42:48.017588 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.017554    2581 generic.go:356] "Generic (PLEG): container finished" podID="eddf28cf-52f6-4e83-b60f-d19f22386920" containerID="f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04" exitCode=0
Sep 17 17:42:48.018040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.017599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699d79656b-87v2t" event={"ID":"eddf28cf-52f6-4e83-b60f-d19f22386920","Type":"ContainerDied","Data":"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"}
Sep 17 17:42:48.018040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.017608    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-699d79656b-87v2t"
Sep 17 17:42:48.018040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.017660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-699d79656b-87v2t" event={"ID":"eddf28cf-52f6-4e83-b60f-d19f22386920","Type":"ContainerDied","Data":"89767dc04891f596d5b81a58c51ca23064b93ff90131f53984a44c29dff66c2e"}
Sep 17 17:42:48.018040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.017682    2581 scope.go:117] "RemoveContainer" containerID="f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"
Sep 17 17:42:48.025451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.025431    2581 scope.go:117] "RemoveContainer" containerID="f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"
Sep 17 17:42:48.025762 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:42:48.025742    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04\": container with ID starting with f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04 not found: ID does not exist" containerID="f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"
Sep 17 17:42:48.025822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.025770    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04"} err="failed to get container status \"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04\": rpc error: code = NotFound desc = could not find container \"f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04\": container with ID starting with f3b63a8ed6a1683076dd74924fc4399e61b1c2c2474e7fd1d3bdeec54fc20a04 not found: ID does not exist"
Sep 17 17:42:48.036628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.036609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:48.040947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:48.040929    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-699d79656b-87v2t"]
Sep 17 17:42:49.359975 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:49.359944    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eddf28cf-52f6-4e83-b60f-d19f22386920" path="/var/lib/kubelet/pods/eddf28cf-52f6-4e83-b60f-d19f22386920/volumes"
Sep 17 17:42:50.895441 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:42:50.895413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lfzxt"
Sep 17 17:43:00.293568 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:00.293523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:00.386772 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:00.386733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:06.588091 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:06.588054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:43:06.591830 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:06.591810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64c4d86f84-ssf8j"
Sep 17 17:43:30.827440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:30.827406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:40.924194 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:40.924153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:41.291483 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:41.291444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:45.993063 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.992985    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"]
Sep 17 17:43:45.993416 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.993273    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eddf28cf-52f6-4e83-b60f-d19f22386920" containerName="console"
Sep 17 17:43:45.993416 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.993288    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eddf28cf-52f6-4e83-b60f-d19f22386920" containerName="console"
Sep 17 17:43:45.993416 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.993345    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eddf28cf-52f6-4e83-b60f-d19f22386920" containerName="console"
Sep 17 17:43:45.995451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.995433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:45.998499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:43:45.998499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:43:45.998706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:43:45.998706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998544    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:43:45.998706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:43:45.998706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:45.998616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jj2s4\""
Sep 17 17:43:46.004300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.004280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:43:46.016050 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.016022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"]
Sep 17 17:43:46.065441 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-serving-certs-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065441 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065678 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2g7\" (UniqueName: \"kubernetes.io/projected/11cdb31e-25a8-4aba-bb14-b87df7083be9-kube-api-access-zh2g7\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065678 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-metrics-client-ca\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065678 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065678 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-federate-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.065820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.065764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167005 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.166955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167005 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2g7\" (UniqueName: \"kubernetes.io/projected/11cdb31e-25a8-4aba-bb14-b87df7083be9-kube-api-access-zh2g7\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167210 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-metrics-client-ca\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167210 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167315 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167460 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-federate-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167549 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.167549 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.167527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-serving-certs-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.168265 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.168235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-metrics-client-ca\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.168977 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.168951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-serving-certs-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.169314 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.169289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.170158 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.170126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.170332 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.170308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-telemeter-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.170495 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.170475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.174941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.174913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11cdb31e-25a8-4aba-bb14-b87df7083be9-federate-client-tls\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.176043 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.176022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2g7\" (UniqueName: \"kubernetes.io/projected/11cdb31e-25a8-4aba-bb14-b87df7083be9-kube-api-access-zh2g7\") pod \"telemeter-client-7c5fd45c48-fq4vn\" (UID: \"11cdb31e-25a8-4aba-bb14-b87df7083be9\") " pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.304542 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.304506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"
Sep 17 17:43:46.583673 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:46.583568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn"]
Sep 17 17:43:46.586677 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:43:46.586634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cdb31e_25a8_4aba_bb14_b87df7083be9.slice/crio-61147ce2b06c7fd6818aa65ba056dc21e1653a99e6450d50db83fe2ee1b2d9d9 WatchSource:0}: Error finding container 61147ce2b06c7fd6818aa65ba056dc21e1653a99e6450d50db83fe2ee1b2d9d9: Status 404 returned error can't find the container with id 61147ce2b06c7fd6818aa65ba056dc21e1653a99e6450d50db83fe2ee1b2d9d9
Sep 17 17:43:47.230207 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:47.230165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn" event={"ID":"11cdb31e-25a8-4aba-bb14-b87df7083be9","Type":"ContainerStarted","Data":"61147ce2b06c7fd6818aa65ba056dc21e1653a99e6450d50db83fe2ee1b2d9d9"}
Sep 17 17:43:49.241026 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.240983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn" event={"ID":"11cdb31e-25a8-4aba-bb14-b87df7083be9","Type":"ContainerStarted","Data":"8569c9e041a31f62d3faa138562144f08f1f17284f3251de996fa5dfa4981a7e"}
Sep 17 17:43:49.241408 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.241055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn" event={"ID":"11cdb31e-25a8-4aba-bb14-b87df7083be9","Type":"ContainerStarted","Data":"c3079528e4489eca63195452a925d3952232d60083e9063db4b82bacc6cc9bcc"}
Sep 17 17:43:49.241408 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.241084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn" event={"ID":"11cdb31e-25a8-4aba-bb14-b87df7083be9","Type":"ContainerStarted","Data":"29ad7bd819b19eb37ddbb9e406488b041803fb38c4558fbead6a9a8801fde2f9"}
Sep 17 17:43:49.278997 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.278938    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c5fd45c48-fq4vn" podStartSLOduration=2.259808584 podStartE2EDuration="4.278921871s" totalImagesPullingTime="2.019113287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:49.272298275 +0000 UTC m=+168.140431237" watchObservedRunningTime="2026-09-17 17:43:49.278921871 +0000 UTC m=+168.147054832"
Sep 17 17:43:49.509803 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.509723    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:43:49.511839 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.511822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.519632 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.519604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:43:49.520250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.520231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kxlr\""
Sep 17 17:43:49.520294 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.520279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:43:49.520389 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.520374    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:43:49.522149 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.522132    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:43:49.529104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.528765    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:43:49.529460 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.529440    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:43:49.530238 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.530222    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:43:49.536202 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.536172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:43:49.554723 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.554695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:43:49.599725 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.599901 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.599901 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.599901 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.599901 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5j7\" (UniqueName: \"kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.600046 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.600046 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.599936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.613589 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.613550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:49.678535 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.678504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:49.700908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.700865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.700908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.700913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701097 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.700939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701097 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.700976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701097 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.700999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701204 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.701131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701317 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.701295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5j7\" (UniqueName: \"kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701792 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.701767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701949 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.701797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.701949 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.701954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.702166 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.702147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.703411 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.703388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.703501 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.703485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.711502 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.711467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5j7\" (UniqueName: \"kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7\") pod \"console-86f4664cf7-lw5qr\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") " pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:49.820788 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:49.820746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:50.092793 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:50.092771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:43:50.094799 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:43:50.094776    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c86c70c_5bd6_4613_9b83_b0dc7fb78c39.slice/crio-fa7f78d8f5e7e5afeb44ce58e9e3b79c30689ee2f1196a407114ce0beaadc710 WatchSource:0}: Error finding container fa7f78d8f5e7e5afeb44ce58e9e3b79c30689ee2f1196a407114ce0beaadc710: Status 404 returned error can't find the container with id fa7f78d8f5e7e5afeb44ce58e9e3b79c30689ee2f1196a407114ce0beaadc710
Sep 17 17:43:50.247074 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:50.247039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f4664cf7-lw5qr" event={"ID":"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39","Type":"ContainerStarted","Data":"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"}
Sep 17 17:43:50.247486 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:50.247095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f4664cf7-lw5qr" event={"ID":"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39","Type":"ContainerStarted","Data":"fa7f78d8f5e7e5afeb44ce58e9e3b79c30689ee2f1196a407114ce0beaadc710"}
Sep 17 17:43:50.267958 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:50.267911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86f4664cf7-lw5qr" podStartSLOduration=1.267890039 podStartE2EDuration="1.267890039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:50.267432014 +0000 UTC m=+169.135564999" watchObservedRunningTime="2026-09-17 17:43:50.267890039 +0000 UTC m=+169.136023001"
Sep 17 17:43:56.166276 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:56.166231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:56.266018 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:56.265976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:59.821086 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:59.821047    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:59.821412 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:59.821140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:43:59.825512 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:43:59.825495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:44:00.285618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:00.285588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:44:00.490035 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:00.490001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.567390 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:00.567313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.800352 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:00.800318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.861679 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:00.861589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.431409 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:01.431376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.493246 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:01.493212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:02.198618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:02.198584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:02.298886 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:02.298855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:08.089810 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:08.089780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:08.156657 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:08.156609    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:11.360789 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:11.360758    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:44:11.469492 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:11.469451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:11.570538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:11.570504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:11.633589 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:11.633509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.291898 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:15.291861    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.393565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:15.393533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:15.493663 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:15.493610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:36.379208 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.379167    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86f4664cf7-lw5qr" podUID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" containerName="console" containerID="cri-o://b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef" gracePeriod=15
Sep 17 17:44:36.625541 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.625519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:44:36.798079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh5j7\" (UniqueName: \"kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798206    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798301 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config\") pod \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\" (UID: \"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39\") "
Sep 17 17:44:36.798605 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:36.798736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798679    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:36.798736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:36.798736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.798712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config" (OuterVolumeSpecName: "console-config") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:36.800237 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.800207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:36.800237 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.800212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7" (OuterVolumeSpecName: "kube-api-access-nh5j7") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "kube-api-access-nh5j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:36.800366 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.800247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" (UID: "4c86c70c-5bd6-4613-9b83-b0dc7fb78c39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:36.899407 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899375    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899407 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899402    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-service-ca\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899407 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899412    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-trusted-ca-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899420    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-oauth-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899430    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-console-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh5j7\" (UniqueName: \"kubernetes.io/projected/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-kube-api-access-nh5j7\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:36.899611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:36.899448    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39-oauth-serving-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:44:37.419815 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.419784    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" containerID="b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef" exitCode=0
Sep 17 17:44:37.420227 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.419825    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f4664cf7-lw5qr"
Sep 17 17:44:37.420227 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.419869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f4664cf7-lw5qr" event={"ID":"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39","Type":"ContainerDied","Data":"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"}
Sep 17 17:44:37.420227 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.419918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f4664cf7-lw5qr" event={"ID":"4c86c70c-5bd6-4613-9b83-b0dc7fb78c39","Type":"ContainerDied","Data":"fa7f78d8f5e7e5afeb44ce58e9e3b79c30689ee2f1196a407114ce0beaadc710"}
Sep 17 17:44:37.420227 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.419938    2581 scope.go:117] "RemoveContainer" containerID="b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"
Sep 17 17:44:37.428330 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.428300    2581 scope.go:117] "RemoveContainer" containerID="b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"
Sep 17 17:44:37.428657 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:44:37.428619    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef\": container with ID starting with b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef not found: ID does not exist" containerID="b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"
Sep 17 17:44:37.428733 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.428662    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef"} err="failed to get container status \"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef\": rpc error: code = NotFound desc = could not find container \"b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef\": container with ID starting with b2569cd02afad768f44dbf04fff2a9e605b877c05743ddfc9166aef5ca7128ef not found: ID does not exist"
Sep 17 17:44:37.438606 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.438586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:44:37.442942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:37.442921    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f4664cf7-lw5qr"]
Sep 17 17:44:39.360556 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:39.360521    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" path="/var/lib/kubelet/pods/4c86c70c-5bd6-4613-9b83-b0dc7fb78c39/volumes"
Sep 17 17:44:42.056714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:42.056670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:42.796951 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:44:42.796907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:14.042182 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:14.042141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:14.143550 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:14.143463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:14.243465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:14.243429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:15.238487 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.238450    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"]
Sep 17 17:45:15.238865 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.238768    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" containerName="console"
Sep 17 17:45:15.238865 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.238782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" containerName="console"
Sep 17 17:45:15.238865 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.238847    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c86c70c-5bd6-4613-9b83-b0dc7fb78c39" containerName="console"
Sep 17 17:45:15.240776 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.240759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.244366 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.244347    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mmkl4\""
Sep 17 17:45:15.244504 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.244378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:45:15.245574 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.245556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:45:15.251229 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.251206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"]
Sep 17 17:45:15.303694 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.303628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.303896 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.303720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.303896 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.303761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcpn\" (UniqueName: \"kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.404516 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.404480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcpn\" (UniqueName: \"kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.404720 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.404557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.404720 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.404591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.404940 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.404921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.404996 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.404947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.413926 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.413898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcpn\" (UniqueName: \"kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.549773 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.549742    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:15.819171 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:15.819144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"]
Sep 17 17:45:15.821706 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:45:15.821677    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aca7a38_7e32_4f7f_bdd0_f4a27f8d2fbc.slice/crio-7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0 WatchSource:0}: Error finding container 7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0: Status 404 returned error can't find the container with id 7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0
Sep 17 17:45:16.559582 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:16.559540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4" event={"ID":"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc","Type":"ContainerStarted","Data":"7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0"}
Sep 17 17:45:22.585777 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:22.585744    2581 generic.go:356] "Generic (PLEG): container finished" podID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerID="6e8c82ffe86f85c998618744cae1d95c44f53ec70c0c252d53f062c90fb2e1ae" exitCode=0
Sep 17 17:45:22.585777 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:22.585780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4" event={"ID":"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc","Type":"ContainerDied","Data":"6e8c82ffe86f85c998618744cae1d95c44f53ec70c0c252d53f062c90fb2e1ae"}
Sep 17 17:45:25.599097 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:25.599061    2581 generic.go:356] "Generic (PLEG): container finished" podID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerID="1e69732a9aedc2a9984b64964cb904558ff5a51c519e3562936542121dd3bd43" exitCode=0
Sep 17 17:45:25.599464 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:25.599127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4" event={"ID":"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc","Type":"ContainerDied","Data":"1e69732a9aedc2a9984b64964cb904558ff5a51c519e3562936542121dd3bd43"}
Sep 17 17:45:33.634521 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:33.634487    2581 generic.go:356] "Generic (PLEG): container finished" podID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerID="75f12a98e3787434d71c298356bd2c6276818dfc0487661fa3f18577324b247a" exitCode=0
Sep 17 17:45:33.634922 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:33.634573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4" event={"ID":"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc","Type":"ContainerDied","Data":"75f12a98e3787434d71c298356bd2c6276818dfc0487661fa3f18577324b247a"}
Sep 17 17:45:34.784695 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.784665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:34.872347 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.872314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util\") pod \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") "
Sep 17 17:45:34.872347 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.872351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jcpn\" (UniqueName: \"kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn\") pod \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") "
Sep 17 17:45:34.872630 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.872380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle\") pod \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\" (UID: \"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc\") "
Sep 17 17:45:34.873008 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.872984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle" (OuterVolumeSpecName: "bundle") pod "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" (UID: "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:34.874563 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.874543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn" (OuterVolumeSpecName: "kube-api-access-2jcpn") pod "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" (UID: "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc"). InnerVolumeSpecName "kube-api-access-2jcpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:34.876707 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.876684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util" (OuterVolumeSpecName: "util") pod "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" (UID: "4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:34.973699 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.973611    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:45:34.973699 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.973657    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcpn\" (UniqueName: \"kubernetes.io/projected/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-kube-api-access-2jcpn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:45:34.973699 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:34.973669    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:45:35.643842 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:35.643767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4" event={"ID":"4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc","Type":"ContainerDied","Data":"7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0"}
Sep 17 17:45:35.643842 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:35.643805    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1av9wx4"
Sep 17 17:45:35.644011 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:35.643812    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab5f0764799b83e71b8f4aa741bc62315474ee8c80ee43fe92808fa156d92e0"
Sep 17 17:45:37.175052 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:37.175006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:37.351213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:37.351176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.138479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.138438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.424042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.423920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.443821 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.443795    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c26n5"]
Sep 17 17:45:44.444129 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="extract"
Sep 17 17:45:44.444213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444131    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="extract"
Sep 17 17:45:44.444213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444147    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="pull"
Sep 17 17:45:44.444213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="pull"
Sep 17 17:45:44.444213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444166    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="util"
Sep 17 17:45:44.444213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444173    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="util"
Sep 17 17:45:44.444444 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.444264    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aca7a38-7e32-4f7f-bdd0-f4a27f8d2fbc" containerName="extract"
Sep 17 17:45:44.446312 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.446293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.448845 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.448824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.448962 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.448928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/08acd1cb-8869-4531-ba4a-55e0e9dfd621-cabundle0\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.449025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.448960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msb9\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-kube-api-access-2msb9\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.449569 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.449547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 17:45:44.449650 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.449566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:45:44.449650 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.449616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:45:44.449755 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.449676    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:45:44.450966 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.450948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:45:44.451021 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.451005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-bt44l\""
Sep 17 17:45:44.458178 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.458155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c26n5"]
Sep 17 17:45:44.549800 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.549759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/08acd1cb-8869-4531-ba4a-55e0e9dfd621-cabundle0\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.549800 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.549802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msb9\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-kube-api-access-2msb9\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.550068 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.549869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.550068 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:44.549997    2581 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 17:45:44.550068 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:44.550016    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:45:44.550068 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:44.550027    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:45:44.550068 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:44.550042    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c26n5: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:45:44.550300 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:44.550109    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates podName:08acd1cb-8869-4531-ba4a-55e0e9dfd621 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:45.050089734 +0000 UTC m=+283.918222698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates") pod "keda-operator-579c8958f7-c26n5" (UID: "08acd1cb-8869-4531-ba4a-55e0e9dfd621") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:45:44.550420 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.550400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/08acd1cb-8869-4531-ba4a-55e0e9dfd621-cabundle0\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.560666 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.560629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msb9\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-kube-api-access-2msb9\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:44.917872 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:44.917831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.013686 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.013634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.054123 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.054094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:45.054238 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:45.054214    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:45:45.054238 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:45.054230    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:45:45.054300 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:45.054239    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c26n5: references non-existent secret key: ca.crt
Sep 17 17:45:45.054333 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:45:45.054305    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates podName:08acd1cb-8869-4531-ba4a-55e0e9dfd621 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:46.054286225 +0000 UTC m=+284.922419167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates") pod "keda-operator-579c8958f7-c26n5" (UID: "08acd1cb-8869-4531-ba4a-55e0e9dfd621") : references non-existent secret key: ca.crt
Sep 17 17:45:45.123920 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.123892    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"]
Sep 17 17:45:45.126833 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.126812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.129866 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.129845    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 17:45:45.141816 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.141791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"]
Sep 17 17:45:45.154534 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.154509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.154667 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.154563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdz6\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-kube-api-access-vpdz6\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.154667 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.154593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.208945 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.208857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.255616 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.255582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.255775 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.255625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdz6\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-kube-api-access-vpdz6\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.255775 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.255671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.256016 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.256000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.258221 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.258203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.282050 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.282023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdz6\" (UniqueName: \"kubernetes.io/projected/0b416fc6-4727-48da-8a93-b8fbb53b7dd3-kube-api-access-vpdz6\") pod \"keda-metrics-apiserver-77857b7b98-wrwps\" (UID: \"0b416fc6-4727-48da-8a93-b8fbb53b7dd3\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.370625 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.370591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.436556 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.436522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:45.459224 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.459169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-bqtx5"]
Sep 17 17:45:45.461968 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.461949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.470420 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.470394    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:45:45.493455 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.493337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-bqtx5"]
Sep 17 17:45:45.557719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.557691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-certificates\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.557912 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.557726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p26k\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-kube-api-access-4p26k\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.573220 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.573190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.658218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.658187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-certificates\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.658419 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.658228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p26k\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-kube-api-access-4p26k\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.660732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.660707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-certificates\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.673302 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.673272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:45.673828 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.673811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p26k\" (UniqueName: \"kubernetes.io/projected/486df3fe-5caf-47d6-b9b0-c249bc048702-kube-api-access-4p26k\") pod \"keda-admission-6c4d864b86-bqtx5\" (UID: \"486df3fe-5caf-47d6-b9b0-c249bc048702\") " pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.719210 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.719135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"]
Sep 17 17:45:45.722098 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:45:45.722070    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b416fc6_4727_48da_8a93_b8fbb53b7dd3.slice/crio-8a58ee5a8b2b6272e61d0bbcb0066811204ed390a65481dd7e0baeaeabd94d5f WatchSource:0}: Error finding container 8a58ee5a8b2b6272e61d0bbcb0066811204ed390a65481dd7e0baeaeabd94d5f: Status 404 returned error can't find the container with id 8a58ee5a8b2b6272e61d0bbcb0066811204ed390a65481dd7e0baeaeabd94d5f
Sep 17 17:45:45.773823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.773788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:45.900420 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:45.900383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:46.053872 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.053849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-bqtx5"]
Sep 17 17:45:46.055784 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:45:46.055760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486df3fe_5caf_47d6_b9b0_c249bc048702.slice/crio-b49d4d6c29c9cb44916736688cfbdf502738017da36613d2a3a6fd7866f49d56 WatchSource:0}: Error finding container b49d4d6c29c9cb44916736688cfbdf502738017da36613d2a3a6fd7866f49d56: Status 404 returned error can't find the container with id b49d4d6c29c9cb44916736688cfbdf502738017da36613d2a3a6fd7866f49d56
Sep 17 17:45:46.060921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.060902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:46.063257 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.063229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/08acd1cb-8869-4531-ba4a-55e0e9dfd621-certificates\") pod \"keda-operator-579c8958f7-c26n5\" (UID: \"08acd1cb-8869-4531-ba4a-55e0e9dfd621\") " pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:46.159434 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.159397    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:46.255822 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.255788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:46.360693 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.360656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:46.459291 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.459256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:46.532107 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.532070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c26n5"]
Sep 17 17:45:46.535543 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:45:46.535515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08acd1cb_8869_4531_ba4a_55e0e9dfd621.slice/crio-37ce0fc2b31a8317f2be3704c7ae6f4f9a17aab8de97cfbee9a20f2f4998e3f9 WatchSource:0}: Error finding container 37ce0fc2b31a8317f2be3704c7ae6f4f9a17aab8de97cfbee9a20f2f4998e3f9: Status 404 returned error can't find the container with id 37ce0fc2b31a8317f2be3704c7ae6f4f9a17aab8de97cfbee9a20f2f4998e3f9
Sep 17 17:45:46.693440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.693335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c26n5" event={"ID":"08acd1cb-8869-4531-ba4a-55e0e9dfd621","Type":"ContainerStarted","Data":"37ce0fc2b31a8317f2be3704c7ae6f4f9a17aab8de97cfbee9a20f2f4998e3f9"}
Sep 17 17:45:46.694994 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.694955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps" event={"ID":"0b416fc6-4727-48da-8a93-b8fbb53b7dd3","Type":"ContainerStarted","Data":"8a58ee5a8b2b6272e61d0bbcb0066811204ed390a65481dd7e0baeaeabd94d5f"}
Sep 17 17:45:46.696564 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:46.696511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5" event={"ID":"486df3fe-5caf-47d6-b9b0-c249bc048702","Type":"ContainerStarted","Data":"b49d4d6c29c9cb44916736688cfbdf502738017da36613d2a3a6fd7866f49d56"}
Sep 17 17:45:48.709079 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.708984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps" event={"ID":"0b416fc6-4727-48da-8a93-b8fbb53b7dd3","Type":"ContainerStarted","Data":"881697d6f9cfe3e6f20e47a48912e1103ffa0f945bc0e2e05ee00d51f861a7a6"}
Sep 17 17:45:48.709493 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.709129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:45:48.710982 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.710949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5" event={"ID":"486df3fe-5caf-47d6-b9b0-c249bc048702","Type":"ContainerStarted","Data":"3d32eb48079961baf03688492e1f2b2f94ce9152d2670adeec98ed4a2fe71eac"}
Sep 17 17:45:48.711117 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.711087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:45:48.732625 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.732571    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps" podStartSLOduration=1.079428996 podStartE2EDuration="3.732551425s" totalImagesPullingTime="2.653122429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:48.730940447 +0000 UTC m=+287.599073411" watchObservedRunningTime="2026-09-17 17:45:48.732551425 +0000 UTC m=+287.600684388"
Sep 17 17:45:48.759457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:48.759403    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5" podStartSLOduration=1.479253694 podStartE2EDuration="3.759389269s" totalImagesPullingTime="2.280135575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:48.758581662 +0000 UTC m=+287.626714626" watchObservedRunningTime="2026-09-17 17:45:48.759389269 +0000 UTC m=+287.627522233"
Sep 17 17:45:50.721040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:50.721005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c26n5" event={"ID":"08acd1cb-8869-4531-ba4a-55e0e9dfd621","Type":"ContainerStarted","Data":"44c247e13b39f04edb5881ca4b1607ae0817a3af7484919225d32d7d3c77e835"}
Sep 17 17:45:50.721428 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:50.721075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:45:50.739912 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:50.739860    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-c26n5" podStartSLOduration=3.33881453 podStartE2EDuration="6.739843405s" totalImagesPullingTime="3.401028875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:50.739082229 +0000 UTC m=+289.607215209" watchObservedRunningTime="2026-09-17 17:45:50.739843405 +0000 UTC m=+289.607976367"
Sep 17 17:45:59.721904 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:45:59.721876    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wrwps"
Sep 17 17:46:03.229333 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:03.229309    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:46:03.236683 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:03.236657    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:46:03.259921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:03.259889    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:46:09.717995 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:09.717968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-bqtx5"
Sep 17 17:46:11.727509 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:11.727476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-c26n5"
Sep 17 17:46:35.925993 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:35.925955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:37.097228 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:46:37.097181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.163451 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.163380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.250499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.250462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.277576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.277546    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-thn7f"]
Sep 17 17:47:07.279952 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.279937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.283257 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.283234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zlkc6\""
Sep 17 17:47:07.283380 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.283279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:47:07.283519 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.283498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:47:07.284770 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.284753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:47:07.293957 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.293935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-thn7f"]
Sep 17 17:47:07.350735 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.350707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.411631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.411586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d997268d-93a5-45e9-94a7-83dda54ad5a6-data\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.411790 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.411728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5977\" (UniqueName: \"kubernetes.io/projected/d997268d-93a5-45e9-94a7-83dda54ad5a6-kube-api-access-z5977\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.512898 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.512815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d997268d-93a5-45e9-94a7-83dda54ad5a6-data\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.512898 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.512871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5977\" (UniqueName: \"kubernetes.io/projected/d997268d-93a5-45e9-94a7-83dda54ad5a6-kube-api-access-z5977\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.513187 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.513167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d997268d-93a5-45e9-94a7-83dda54ad5a6-data\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.522903 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.522883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5977\" (UniqueName: \"kubernetes.io/projected/d997268d-93a5-45e9-94a7-83dda54ad5a6-kube-api-access-z5977\") pod \"seaweedfs-64568bcd49-thn7f\" (UID: \"d997268d-93a5-45e9-94a7-83dda54ad5a6\") " pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.563876 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.563844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.588724 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.588697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:07.665971 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.665915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.835734 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.835694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:07.862875 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.862847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-thn7f"]
Sep 17 17:47:07.866231 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:47:07.866205    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd997268d_93a5_45e9_94a7_83dda54ad5a6.slice/crio-e852269ca49380e3df29222315235836676e2d8931a7bfdac2fda7bc1e3bd8cf WatchSource:0}: Error finding container e852269ca49380e3df29222315235836676e2d8931a7bfdac2fda7bc1e3bd8cf: Status 404 returned error can't find the container with id e852269ca49380e3df29222315235836676e2d8931a7bfdac2fda7bc1e3bd8cf
Sep 17 17:47:07.867506 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.867485    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:07.937176 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:07.937141    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:08.008446 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:08.008417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:08.020422 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:08.020395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-thn7f" event={"ID":"d997268d-93a5-45e9-94a7-83dda54ad5a6","Type":"ContainerStarted","Data":"e852269ca49380e3df29222315235836676e2d8931a7bfdac2fda7bc1e3bd8cf"}
Sep 17 17:47:10.936108 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.936075    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9fd7587ff-8plxx"]
Sep 17 17:47:10.938498 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.938478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:10.944809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.944788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kxlr\""
Sep 17 17:47:10.945104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.945085    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:47:10.945104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.945103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:47:10.945255 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.945086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:47:10.945255 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.945134    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:47:10.946609 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.946592    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:47:10.946872 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.946855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:47:10.946948 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.946919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:47:10.949573 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.949555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:47:10.955003 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:10.954958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fd7587ff-8plxx"]
Sep 17 17:47:11.002314 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.002289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:11.033374 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.033348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-thn7f" event={"ID":"d997268d-93a5-45e9-94a7-83dda54ad5a6","Type":"ContainerStarted","Data":"25c87db573d18f6d52e50d4ec46821af7de1af744be4cd39b0ea3584ea45a168"}
Sep 17 17:47:11.033499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.033473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:11.045843 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-oauth-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.045942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-oauth-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.045942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.045942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8dk\" (UniqueName: \"kubernetes.io/projected/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-kube-api-access-rl8dk\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.046042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-service-ca\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.046042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.045983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.046042 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.046005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-trusted-ca-bundle\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.052136 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.052089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-thn7f" podStartSLOduration=1.213398747 podStartE2EDuration="4.052072651s" totalImagesPullingTime="2.838673904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:11.051929568 +0000 UTC m=+369.920062557" watchObservedRunningTime="2026-09-17 17:47:11.052072651 +0000 UTC m=+369.920205615"
Sep 17 17:47:11.101687 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.101661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:11.146804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.146915 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-trusted-ca-bundle\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.146915 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-oauth-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.146915 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-oauth-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.146948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8dk\" (UniqueName: \"kubernetes.io/projected/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-kube-api-access-rl8dk\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.147016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-service-ca\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147743 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.147715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-oauth-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147855 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.147768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147904 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.147851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-service-ca\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.147904 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.147876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-trusted-ca-bundle\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.149397 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.149375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-oauth-config\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.149536 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.149518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-console-serving-cert\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.155541 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.155522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8dk\" (UniqueName: \"kubernetes.io/projected/1bda909b-9cfe-4cfb-ab16-8e4c913b035a-kube-api-access-rl8dk\") pod \"console-9fd7587ff-8plxx\" (UID: \"1bda909b-9cfe-4cfb-ab16-8e4c913b035a\") " pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.247791 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.247722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:11.518418 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:11.518390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fd7587ff-8plxx"]
Sep 17 17:47:11.520142 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:47:11.520118    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bda909b_9cfe_4cfb_ab16_8e4c913b035a.slice/crio-fcfeebcc23fdffe7cf9aa451f37e1a1a4e9fbbb7e07ca28f8d7f737866723148 WatchSource:0}: Error finding container fcfeebcc23fdffe7cf9aa451f37e1a1a4e9fbbb7e07ca28f8d7f737866723148: Status 404 returned error can't find the container with id fcfeebcc23fdffe7cf9aa451f37e1a1a4e9fbbb7e07ca28f8d7f737866723148
Sep 17 17:47:12.038708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:12.038669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fd7587ff-8plxx" event={"ID":"1bda909b-9cfe-4cfb-ab16-8e4c913b035a","Type":"ContainerStarted","Data":"5508cbea8b66894b12be057f156166c4248fa84eea6fa9f7a096f7432fea9a03"}
Sep 17 17:47:12.038708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:12.038726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fd7587ff-8plxx" event={"ID":"1bda909b-9cfe-4cfb-ab16-8e4c913b035a","Type":"ContainerStarted","Data":"fcfeebcc23fdffe7cf9aa451f37e1a1a4e9fbbb7e07ca28f8d7f737866723148"}
Sep 17 17:47:12.058553 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:12.058506    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9fd7587ff-8plxx" podStartSLOduration=2.05849001 podStartE2EDuration="2.05849001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:12.058381596 +0000 UTC m=+370.926514560" watchObservedRunningTime="2026-09-17 17:47:12.05849001 +0000 UTC m=+370.926622977"
Sep 17 17:47:17.041331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:17.041300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-thn7f"
Sep 17 17:47:21.248123 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:21.248088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:21.248497 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:21.248179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:21.253010 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:21.252993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:22.082296 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:22.082266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9fd7587ff-8plxx"
Sep 17 17:47:22.371759 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:22.371680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:22.480718 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:22.480682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:53.843225 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:53.843183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:53.945326 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:47:53.945291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:09.360111 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:09.360077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:25.850896 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:25.850797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:25.972098 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:25.972063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:26.144720 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:26.144610    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:26.244655 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:26.244617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:27.324617 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:27.324581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:27.407456 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:27.407419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:47.191576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:48:47.191538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:07.208230 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:49:07.208189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:31.557802 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:49:31.557759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:01.929356 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:01.929309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:02.054006 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:02.053968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:14.967663 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:14.967611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:15.302003 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.301967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:15.395805 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.395774    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:50:15.398544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.398524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.404156 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.404131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 17:50:15.404956 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.404937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:50:15.405541 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.405523    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:50:15.405711 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.405525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:50:15.405711 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.405597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:50:15.405711 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.405613    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:50:15.405926 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.405911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:50:15.422479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.422456    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:50:15.425440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.425421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:50:15.425529 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.425513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.428275 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.428256    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:50:15.431731 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.431708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:15.440596 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.440574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:50:15.467200 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.467315 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.467315 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrjc\" (UniqueName: \"kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.467315 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.467479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.467479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.467479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.467618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkpf\" (UniqueName: \"kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.467618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.467618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.467553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.568267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkpf\" (UniqueName: \"kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568511 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568834 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrjc\" (UniqueName: \"kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568834 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.568834 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.568698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.569131 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.569107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.569131 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.569127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.569322 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.569298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.569570 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.569551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.570964 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.570940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.571040 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.571019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.581619 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.581598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrjc\" (UniqueName: \"kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc\") pod \"isvc-canary-rollback-predictor-76cbc5464-2bd86\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.581867 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.581850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkpf\" (UniqueName: \"kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-5djmv\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.584099 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.584066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:15.707662 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.707613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:15.734965 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.734930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:15.980560 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:15.980532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:50:15.981823 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:50:15.981798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886ef972_9ff0_4f2d_b166_b062ad445941.slice/crio-66c7e79d9768f4cf1375b544723ee444af42ea93ecd6d077804caa390b156d21 WatchSource:0}: Error finding container 66c7e79d9768f4cf1375b544723ee444af42ea93ecd6d077804caa390b156d21: Status 404 returned error can't find the container with id 66c7e79d9768f4cf1375b544723ee444af42ea93ecd6d077804caa390b156d21
Sep 17 17:50:16.026823 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:16.026796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:50:16.029347 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:50:16.029322    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307cc23e_31b8_4304_bd04_4883e9dd1273.slice/crio-62129a6c6a6b552e357c08a001c0287751a3d1876ab553a3b7cde481eeece7af WatchSource:0}: Error finding container 62129a6c6a6b552e357c08a001c0287751a3d1876ab553a3b7cde481eeece7af: Status 404 returned error can't find the container with id 62129a6c6a6b552e357c08a001c0287751a3d1876ab553a3b7cde481eeece7af
Sep 17 17:50:16.750942 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:16.750847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerStarted","Data":"62129a6c6a6b552e357c08a001c0287751a3d1876ab553a3b7cde481eeece7af"}
Sep 17 17:50:16.752628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:16.752548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerStarted","Data":"66c7e79d9768f4cf1375b544723ee444af42ea93ecd6d077804caa390b156d21"}
Sep 17 17:50:19.769179 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:19.769130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerStarted","Data":"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"}
Sep 17 17:50:19.771031 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:19.771000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerStarted","Data":"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"}
Sep 17 17:50:23.790223 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:23.790187    2581 generic.go:356] "Generic (PLEG): container finished" podID="886ef972-9ff0-4f2d-b166-b062ad445941" containerID="20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4" exitCode=0
Sep 17 17:50:23.790675 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:23.790261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerDied","Data":"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"}
Sep 17 17:50:23.793248 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:23.793217    2581 generic.go:356] "Generic (PLEG): container finished" podID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerID="1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391" exitCode=0
Sep 17 17:50:23.793399 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:23.793380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerDied","Data":"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"}
Sep 17 17:50:24.118258 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:24.118158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:33.295447 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:33.295402    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:39.891524 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:39.891491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerStarted","Data":"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"}
Sep 17 17:50:39.893162 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:39.893135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerStarted","Data":"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"}
Sep 17 17:50:42.909455 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.909414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerStarted","Data":"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"}
Sep 17 17:50:42.909921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.909590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:42.909921 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.909632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:42.911231 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.911188    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:42.911771 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.911750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerStarted","Data":"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"}
Sep 17 17:50:42.911890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.911873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:42.938856 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.938811    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podStartSLOduration=2.357231176 podStartE2EDuration="27.938798083s" totalImagesPullingTime="21.778595919s" totalInitContainerRuntime="3.802970988s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:42.937035444 +0000 UTC m=+581.805168417" watchObservedRunningTime="2026-09-17 17:50:42.938798083 +0000 UTC m=+581.806931083"
Sep 17 17:50:42.968397 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:42.968351    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podStartSLOduration=2.49114762 podStartE2EDuration="27.968338614s" totalImagesPullingTime="21.7303986s" totalInitContainerRuntime="3.746792394s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:42.967868697 +0000 UTC m=+581.836001660" watchObservedRunningTime="2026-09-17 17:50:42.968338614 +0000 UTC m=+581.836471577"
Sep 17 17:50:43.916159 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:43.916111    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:43.916674 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:43.916170    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:43.917379 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:43.917358    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:44.920365 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:44.920325    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:48.920660 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:48.920617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:50:48.921200 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:48.921161    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:49.924929 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:49.924896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:50:49.925432 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:49.925408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:58.921261 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:58.921224    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:59.925858 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:50:59.925822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:03.263341 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:03.263316    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:51:03.289073 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:03.289051    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:51:08.921962 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:08.921921    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:09.925524 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:09.925488    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:17.224479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:17.224392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:17.327577 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:17.327535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:18.921274 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:18.921237    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:19.925442 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:19.925402    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:28.921454 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:28.921409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:29.925386 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:29.925350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:38.922022 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:38.921984    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:39.926135 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:39.926100    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:48.921807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:48.921768    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:49.925869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:49.925829    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:51.356616 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:51.356570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:58.921817 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:58.921786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:51:59.053996 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.053960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:51:59.054230 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.054180    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container" containerID="cri-o://a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35" gracePeriod=30
Sep 17 17:51:59.054369 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.054249    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kube-rbac-proxy" containerID="cri-o://704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87" gracePeriod=30
Sep 17 17:51:59.055458 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.055439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:51:59.225247 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.225161    2581 generic.go:356] "Generic (PLEG): container finished" podID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerID="704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87" exitCode=2
Sep 17 17:51:59.225247 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.225227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerDied","Data":"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"}
Sep 17 17:51:59.920875 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:51:59.920833    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 17:52:00.050637 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.050597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:00.246851 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.246769    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:52:00.249350 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.249329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.252316 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.252289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-bbc14-predictor-serving-cert\""
Sep 17 17:52:00.252459 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.252299    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-bbc14-kube-rbac-proxy-sar-config\""
Sep 17 17:52:00.264196 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.264176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:52:00.324063 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.324034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.324218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.324069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.324218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.324089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmnz\" (UniqueName: \"kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.324218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.324130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.324218 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.324184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.425538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.425504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.425749 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.425546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.425749 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.425591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.425749 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.425613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.425749 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.425632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmnz\" (UniqueName: \"kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.426085 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.426047    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.426335 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.426311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.426411 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.426335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.428110 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.428092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.434839 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.434820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmnz\" (UniqueName: \"kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz\") pod \"isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.559317 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.559283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:00.831982 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:00.831958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:52:00.836205 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:52:00.836139    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee8a02f_0c12_4a6e_bb53_df9a11964b06.slice/crio-f2dfdfc65aeea203ce6629b25a6f0ea6e6c46d60fb7605289db25cf98d5849da WatchSource:0}: Error finding container f2dfdfc65aeea203ce6629b25a6f0ea6e6c46d60fb7605289db25cf98d5849da: Status 404 returned error can't find the container with id f2dfdfc65aeea203ce6629b25a6f0ea6e6c46d60fb7605289db25cf98d5849da
Sep 17 17:52:01.235499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:01.235421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerStarted","Data":"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"}
Sep 17 17:52:01.235499 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:01.235502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerStarted","Data":"f2dfdfc65aeea203ce6629b25a6f0ea6e6c46d60fb7605289db25cf98d5849da"}
Sep 17 17:52:01.577027 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:01.576997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:03.805868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.805840    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:52:03.960028 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.959935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert\") pod \"307cc23e-31b8-4304-bd04-4883e9dd1273\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") "
Sep 17 17:52:03.960028 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.959990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxrjc\" (UniqueName: \"kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc\") pod \"307cc23e-31b8-4304-bd04-4883e9dd1273\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") "
Sep 17 17:52:03.960250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960076    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls\") pod \"307cc23e-31b8-4304-bd04-4883e9dd1273\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") "
Sep 17 17:52:03.960250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location\") pod \"307cc23e-31b8-4304-bd04-4883e9dd1273\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") "
Sep 17 17:52:03.960250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config\") pod \"307cc23e-31b8-4304-bd04-4883e9dd1273\" (UID: \"307cc23e-31b8-4304-bd04-4883e9dd1273\") "
Sep 17 17:52:03.960465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "307cc23e-31b8-4304-bd04-4883e9dd1273" (UID: "307cc23e-31b8-4304-bd04-4883e9dd1273"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:03.960526 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "307cc23e-31b8-4304-bd04-4883e9dd1273" (UID: "307cc23e-31b8-4304-bd04-4883e9dd1273"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:03.960568 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.960523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "307cc23e-31b8-4304-bd04-4883e9dd1273" (UID: "307cc23e-31b8-4304-bd04-4883e9dd1273"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:03.962264 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.962241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc" (OuterVolumeSpecName: "kube-api-access-vxrjc") pod "307cc23e-31b8-4304-bd04-4883e9dd1273" (UID: "307cc23e-31b8-4304-bd04-4883e9dd1273"). InnerVolumeSpecName "kube-api-access-vxrjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:03.962343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:03.962256    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "307cc23e-31b8-4304-bd04-4883e9dd1273" (UID: "307cc23e-31b8-4304-bd04-4883e9dd1273"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:04.061069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.061027    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrjc\" (UniqueName: \"kubernetes.io/projected/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-api-access-vxrjc\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:04.061069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.061061    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307cc23e-31b8-4304-bd04-4883e9dd1273-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:04.061069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.061072    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307cc23e-31b8-4304-bd04-4883e9dd1273-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:04.061069 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.061081    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:04.061334 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.061090    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307cc23e-31b8-4304-bd04-4883e9dd1273-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:04.249824 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.249743    2581 generic.go:356] "Generic (PLEG): container finished" podID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerID="a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35" exitCode=0
Sep 17 17:52:04.249824 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.249802    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"
Sep 17 17:52:04.250002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.249801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerDied","Data":"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"}
Sep 17 17:52:04.250002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.249929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86" event={"ID":"307cc23e-31b8-4304-bd04-4883e9dd1273","Type":"ContainerDied","Data":"62129a6c6a6b552e357c08a001c0287751a3d1876ab553a3b7cde481eeece7af"}
Sep 17 17:52:04.250002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.249960    2581 scope.go:117] "RemoveContainer" containerID="704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"
Sep 17 17:52:04.258512 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.258496    2581 scope.go:117] "RemoveContainer" containerID="a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"
Sep 17 17:52:04.266002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.265984    2581 scope.go:117] "RemoveContainer" containerID="1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"
Sep 17 17:52:04.273710 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.273628    2581 scope.go:117] "RemoveContainer" containerID="704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"
Sep 17 17:52:04.274051 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:04.274030    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87\": container with ID starting with 704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87 not found: ID does not exist" containerID="704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"
Sep 17 17:52:04.274136 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.274060    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87"} err="failed to get container status \"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87\": rpc error: code = NotFound desc = could not find container \"704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87\": container with ID starting with 704c3e4df49035e19345e52b40079faab4065dd6c6a06c9e2d13a5f0992b4f87 not found: ID does not exist"
Sep 17 17:52:04.274136 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.274081    2581 scope.go:117] "RemoveContainer" containerID="a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"
Sep 17 17:52:04.274349 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:04.274328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35\": container with ID starting with a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35 not found: ID does not exist" containerID="a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"
Sep 17 17:52:04.274412 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.274354    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35"} err="failed to get container status \"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35\": rpc error: code = NotFound desc = could not find container \"a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35\": container with ID starting with a5a8ee7b3b54df037e53650a63468005a80497a4fa4573d055118a9208db7f35 not found: ID does not exist"
Sep 17 17:52:04.274412 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.274374    2581 scope.go:117] "RemoveContainer" containerID="1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"
Sep 17 17:52:04.274624 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:04.274609    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391\": container with ID starting with 1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391 not found: ID does not exist" containerID="1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"
Sep 17 17:52:04.274755 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.274629    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391"} err="failed to get container status \"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391\": rpc error: code = NotFound desc = could not find container \"1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391\": container with ID starting with 1aa33ab53ebbcf3ce8744c3749c4044b4a5eacf565947971f3c3fd829430e391 not found: ID does not exist"
Sep 17 17:52:04.276328 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.276308    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:52:04.277783 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.277765    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-2bd86"]
Sep 17 17:52:04.969543 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.969458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:52:04.969963 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.969713    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" containerID="cri-o://d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd" gracePeriod=30
Sep 17 17:52:04.969963 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:04.969776    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kube-rbac-proxy" containerID="cri-o://f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482" gracePeriod=30
Sep 17 17:52:05.004385 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.004352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:05.026838 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.026808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:52:05.027146 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027135    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kube-rbac-proxy"
Sep 17 17:52:05.027201 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027148    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kube-rbac-proxy"
Sep 17 17:52:05.027201 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027171    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="storage-initializer"
Sep 17 17:52:05.027201 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027176    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="storage-initializer"
Sep 17 17:52:05.027201 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027183    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container"
Sep 17 17:52:05.027201 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027189    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container"
Sep 17 17:52:05.027345 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027238    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kube-rbac-proxy"
Sep 17 17:52:05.027345 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.027247    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" containerName="kserve-container"
Sep 17 17:52:05.029407 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.029388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.032488 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.032457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:52:05.042158 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.042134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:52:05.174704 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.174665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.174881 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.174710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.174881 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.174802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.174881 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.174855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklp5\" (UniqueName: \"kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.175035 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.174954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.256627 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.256543    2581 generic.go:356] "Generic (PLEG): container finished" podID="886ef972-9ff0-4f2d-b166-b062ad445941" containerID="f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482" exitCode=2
Sep 17 17:52:05.256627 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.256615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerDied","Data":"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"}
Sep 17 17:52:05.258081 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.258059    2581 generic.go:356] "Generic (PLEG): container finished" podID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerID="8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4" exitCode=0
Sep 17 17:52:05.258212 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.258142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerDied","Data":"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"}
Sep 17 17:52:05.275491 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.275468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.275579 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.275507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklp5\" (UniqueName: \"kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.275579 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.275537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.275579 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.275562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.275716 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.275692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.276075 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.276042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.276185 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.276142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.276250 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.276205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.277920 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.277902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.285270 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.285248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklp5\" (UniqueName: \"kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5\") pod \"isvc-canary-rollback-predictor-76cbc5464-r5clr\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.339304 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.339266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:05.362839 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.362809    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="307cc23e-31b8-4304-bd04-4883e9dd1273" path="/var/lib/kubelet/pods/307cc23e-31b8-4304-bd04-4883e9dd1273/volumes"
Sep 17 17:52:05.616909 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:05.616878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:52:05.619031 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:52:05.619002    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bff906_999f_4efc_b831_26c77b30b7ec.slice/crio-424e9b42a9c4e5d55b5d725521d03bf3489a8c8072f12f40c5c4242ddbe7a33a WatchSource:0}: Error finding container 424e9b42a9c4e5d55b5d725521d03bf3489a8c8072f12f40c5c4242ddbe7a33a: Status 404 returned error can't find the container with id 424e9b42a9c4e5d55b5d725521d03bf3489a8c8072f12f40c5c4242ddbe7a33a
Sep 17 17:52:06.267491 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:06.267450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerStarted","Data":"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"}
Sep 17 17:52:06.267941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:06.267528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerStarted","Data":"424e9b42a9c4e5d55b5d725521d03bf3489a8c8072f12f40c5c4242ddbe7a33a"}
Sep 17 17:52:08.917065 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:08.917025    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 17:52:08.921937 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:08.921909    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:52:10.212857 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.212794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:10.244452 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.244417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:52:10.294941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.294909    2581 generic.go:356] "Generic (PLEG): container finished" podID="08bff906-999f-4efc-b831-26c77b30b7ec" containerID="b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d" exitCode=0
Sep 17 17:52:10.295093 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.294978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerDied","Data":"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"}
Sep 17 17:52:10.297116 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.297095    2581 generic.go:356] "Generic (PLEG): container finished" podID="886ef972-9ff0-4f2d-b166-b062ad445941" containerID="d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd" exitCode=0
Sep 17 17:52:10.297222 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.297129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerDied","Data":"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"}
Sep 17 17:52:10.297222 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.297149    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"
Sep 17 17:52:10.297222 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.297199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv" event={"ID":"886ef972-9ff0-4f2d-b166-b062ad445941","Type":"ContainerDied","Data":"66c7e79d9768f4cf1375b544723ee444af42ea93ecd6d077804caa390b156d21"}
Sep 17 17:52:10.297331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.297249    2581 scope.go:117] "RemoveContainer" containerID="f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"
Sep 17 17:52:10.308119 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.308089    2581 scope.go:117] "RemoveContainer" containerID="d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"
Sep 17 17:52:10.321976 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.321958    2581 scope.go:117] "RemoveContainer" containerID="20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"
Sep 17 17:52:10.322239 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config\") pod \"886ef972-9ff0-4f2d-b166-b062ad445941\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") "
Sep 17 17:52:10.322311 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location\") pod \"886ef972-9ff0-4f2d-b166-b062ad445941\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") "
Sep 17 17:52:10.322364 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert\") pod \"886ef972-9ff0-4f2d-b166-b062ad445941\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") "
Sep 17 17:52:10.322364 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322354    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls\") pod \"886ef972-9ff0-4f2d-b166-b062ad445941\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") "
Sep 17 17:52:10.322603 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljkpf\" (UniqueName: \"kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf\") pod \"886ef972-9ff0-4f2d-b166-b062ad445941\" (UID: \"886ef972-9ff0-4f2d-b166-b062ad445941\") "
Sep 17 17:52:10.322603 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "886ef972-9ff0-4f2d-b166-b062ad445941" (UID: "886ef972-9ff0-4f2d-b166-b062ad445941"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:10.322754 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "886ef972-9ff0-4f2d-b166-b062ad445941" (UID: "886ef972-9ff0-4f2d-b166-b062ad445941"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:10.322754 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.322717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "886ef972-9ff0-4f2d-b166-b062ad445941" (UID: "886ef972-9ff0-4f2d-b166-b062ad445941"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:10.323088 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.323071    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:10.323180 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.323095    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886ef972-9ff0-4f2d-b166-b062ad445941-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:10.323180 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.323110    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886ef972-9ff0-4f2d-b166-b062ad445941-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:10.324284 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.324263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "886ef972-9ff0-4f2d-b166-b062ad445941" (UID: "886ef972-9ff0-4f2d-b166-b062ad445941"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:10.324576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.324559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf" (OuterVolumeSpecName: "kube-api-access-ljkpf") pod "886ef972-9ff0-4f2d-b166-b062ad445941" (UID: "886ef972-9ff0-4f2d-b166-b062ad445941"). InnerVolumeSpecName "kube-api-access-ljkpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:10.337885 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.337861    2581 scope.go:117] "RemoveContainer" containerID="f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"
Sep 17 17:52:10.338244 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:10.338225    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482\": container with ID starting with f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482 not found: ID does not exist" containerID="f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"
Sep 17 17:52:10.338340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.338251    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482"} err="failed to get container status \"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482\": rpc error: code = NotFound desc = could not find container \"f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482\": container with ID starting with f3b0c9aa9e203131e0dc150198a15cd9de37fb20439c689d2632ca5337cd6482 not found: ID does not exist"
Sep 17 17:52:10.338340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.338269    2581 scope.go:117] "RemoveContainer" containerID="d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"
Sep 17 17:52:10.338705 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:10.338556    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd\": container with ID starting with d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd not found: ID does not exist" containerID="d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"
Sep 17 17:52:10.338705 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.338587    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd"} err="failed to get container status \"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd\": rpc error: code = NotFound desc = could not find container \"d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd\": container with ID starting with d7abd440b88671536f96e72826716371cdebf6dfcbd9d9a7baa8c0228cb5a2fd not found: ID does not exist"
Sep 17 17:52:10.338705 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.338610    2581 scope.go:117] "RemoveContainer" containerID="20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"
Sep 17 17:52:10.339139 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:52:10.339112    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4\": container with ID starting with 20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4 not found: ID does not exist" containerID="20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"
Sep 17 17:52:10.339247 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.339143    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4"} err="failed to get container status \"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4\": rpc error: code = NotFound desc = could not find container \"20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4\": container with ID starting with 20d12430d6c80c2878162710e6168c78c54e44a894071de006ede98924d804d4 not found: ID does not exist"
Sep 17 17:52:10.424510 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.424481    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886ef972-9ff0-4f2d-b166-b062ad445941-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:10.424510 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.424510    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkpf\" (UniqueName: \"kubernetes.io/projected/886ef972-9ff0-4f2d-b166-b062ad445941-kube-api-access-ljkpf\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:52:10.628401 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.628357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:52:10.635088 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:10.635055    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-5djmv"]
Sep 17 17:52:11.305631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.305580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerStarted","Data":"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"}
Sep 17 17:52:11.306117 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.305714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerStarted","Data":"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"}
Sep 17 17:52:11.306117 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.306107    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:11.306275 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.306250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:11.307923 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.307896    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:11.330461 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.330406    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podStartSLOduration=2.489107043 podStartE2EDuration="6.330387578s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841280535s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:11.32903049 +0000 UTC m=+670.197163453" watchObservedRunningTime="2026-09-17 17:52:11.330387578 +0000 UTC m=+670.198520541"
Sep 17 17:52:11.362767 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:11.362726    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" path="/var/lib/kubelet/pods/886ef972-9ff0-4f2d-b166-b062ad445941/volumes"
Sep 17 17:52:12.313401 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:12.313352    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:17.318168 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:17.318136    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:52:17.318884 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:17.318846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:26.384299 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:26.384264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerStarted","Data":"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"}
Sep 17 17:52:26.384719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:26.384325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerStarted","Data":"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"}
Sep 17 17:52:26.384719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:26.384513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:26.410432 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:26.410386    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podStartSLOduration=2.469972951 podStartE2EDuration="26.41037291s" totalImagesPullingTime="20.206511449s" totalInitContainerRuntime="3.73388851s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:26.408844726 +0000 UTC m=+685.276977688" watchObservedRunningTime="2026-09-17 17:52:26.41037291 +0000 UTC m=+685.278505872"
Sep 17 17:52:27.319598 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:27.319555    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:27.389429 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:27.389403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:27.390613 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:27.390585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:52:28.393699 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:28.393659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:52:33.398676 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:33.398627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:52:33.399143 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:33.399117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:52:37.318873 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:37.318833    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:43.399469 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:43.399419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:52:47.319230 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:47.319153    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:53.399366 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:53.399328    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:52:57.319266 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:52:57.319222    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:53:03.399375 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:03.399340    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:53:04.427863 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:04.427824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:07.319486 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:07.319421    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:53:13.399130 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:13.399089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:53:17.319009 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:17.318972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:53:22.355946 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:22.355896    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:53:23.399677 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:23.399619    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:53:32.357746 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:32.357709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:53:33.400804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:33.400770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:53:35.133439 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.133402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:53:35.133982 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.133836    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container" containerID="cri-o://7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425" gracePeriod=30
Sep 17 17:53:35.133982 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.133875    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kube-rbac-proxy" containerID="cri-o://fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178" gracePeriod=30
Sep 17 17:53:35.155035 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.155005    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:35.680455 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.680421    2581 generic.go:356] "Generic (PLEG): container finished" podID="08bff906-999f-4efc-b831-26c77b30b7ec" containerID="fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178" exitCode=2
Sep 17 17:53:35.680618 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:35.680493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerDied","Data":"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"}
Sep 17 17:53:37.313705 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:37.313658    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 17:53:40.081008 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.080979    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:53:40.112608 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location\") pod \"08bff906-999f-4efc-b831-26c77b30b7ec\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") "
Sep 17 17:53:40.112608 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config\") pod \"08bff906-999f-4efc-b831-26c77b30b7ec\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") "
Sep 17 17:53:40.112869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls\") pod \"08bff906-999f-4efc-b831-26c77b30b7ec\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") "
Sep 17 17:53:40.112869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nklp5\" (UniqueName: \"kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5\") pod \"08bff906-999f-4efc-b831-26c77b30b7ec\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") "
Sep 17 17:53:40.112869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert\") pod \"08bff906-999f-4efc-b831-26c77b30b7ec\" (UID: \"08bff906-999f-4efc-b831-26c77b30b7ec\") "
Sep 17 17:53:40.113012 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.112867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08bff906-999f-4efc-b831-26c77b30b7ec" (UID: "08bff906-999f-4efc-b831-26c77b30b7ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:40.113095 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.113006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08bff906-999f-4efc-b831-26c77b30b7ec" (UID: "08bff906-999f-4efc-b831-26c77b30b7ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:40.113152 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.113095    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08bff906-999f-4efc-b831-26c77b30b7ec-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.113206 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.113182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08bff906-999f-4efc-b831-26c77b30b7ec" (UID: "08bff906-999f-4efc-b831-26c77b30b7ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:40.115198 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.115175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08bff906-999f-4efc-b831-26c77b30b7ec" (UID: "08bff906-999f-4efc-b831-26c77b30b7ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:40.115302 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.115223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5" (OuterVolumeSpecName: "kube-api-access-nklp5") pod "08bff906-999f-4efc-b831-26c77b30b7ec" (UID: "08bff906-999f-4efc-b831-26c77b30b7ec"). InnerVolumeSpecName "kube-api-access-nklp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:40.158265 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.158228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.213761 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.213732    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.213761 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.213758    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08bff906-999f-4efc-b831-26c77b30b7ec-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.213761 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.213767    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nklp5\" (UniqueName: \"kubernetes.io/projected/08bff906-999f-4efc-b831-26c77b30b7ec-kube-api-access-nklp5\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.213985 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.213776    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08bff906-999f-4efc-b831-26c77b30b7ec-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.260810 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.260775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.340270 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340228    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:53:40.340668 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="storage-initializer"
Sep 17 17:53:40.340668 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="storage-initializer"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340681    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kube-rbac-proxy"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kube-rbac-proxy"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="storage-initializer"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340717    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="storage-initializer"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340726    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340743    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kube-rbac-proxy"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340750    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kube-rbac-proxy"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container"
Sep 17 17:53:40.340809 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340785    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container"
Sep 17 17:53:40.341267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340865    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kube-rbac-proxy"
Sep 17 17:53:40.341267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340878    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kserve-container"
Sep 17 17:53:40.341267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340889    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="886ef972-9ff0-4f2d-b166-b062ad445941" containerName="kube-rbac-proxy"
Sep 17 17:53:40.341267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.340901    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" containerName="kserve-container"
Sep 17 17:53:40.343365 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.343342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.346817 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.346798    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:53:40.346920 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.346795    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 17:53:40.351976 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.351931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.359477 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.359453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:53:40.403910 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.403833    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:53:40.406820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.406796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.410677 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.410656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:53:40.416030 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.415999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ljs\" (UniqueName: \"kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.416148 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.416148 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.416148 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.416376 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.416321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szz2t\" (UniqueName: \"kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.433944 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.433920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:53:40.506710 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.506669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.516812 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ljs\" (UniqueName: \"kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.516893 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.516893 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.516972 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.516972 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.516972 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.516965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517115 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.517115 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.517216 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517216 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szz2t\" (UniqueName: \"kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517320 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:40.517109    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 17:53:40.517320 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:40.517303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls podName:3d40d9d7-30e9-4745-a158-c4c07d80e51d nodeName:}" failed. No retries permitted until 2026-09-17 17:53:41.017275717 +0000 UTC m=+759.885408669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls") pod "isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 17:53:40.517454 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.517762 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517836 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.517936 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.517998 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.517981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.519421 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.519400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.527612 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.527587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ljs\" (UniqueName: \"kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-z9bmk\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.527869 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.527853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szz2t\" (UniqueName: \"kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:40.619558 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.619521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.703134 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.703051    2581 generic.go:356] "Generic (PLEG): container finished" podID="08bff906-999f-4efc-b831-26c77b30b7ec" containerID="7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425" exitCode=0
Sep 17 17:53:40.703134 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.703109    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"
Sep 17 17:53:40.703307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.703130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerDied","Data":"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"}
Sep 17 17:53:40.703307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.703208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr" event={"ID":"08bff906-999f-4efc-b831-26c77b30b7ec","Type":"ContainerDied","Data":"424e9b42a9c4e5d55b5d725521d03bf3489a8c8072f12f40c5c4242ddbe7a33a"}
Sep 17 17:53:40.703307 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.703247    2581 scope.go:117] "RemoveContainer" containerID="fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"
Sep 17 17:53:40.711332 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.711313    2581 scope.go:117] "RemoveContainer" containerID="7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"
Sep 17 17:53:40.716665 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.716620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:40.718793 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.718769    2581 scope.go:117] "RemoveContainer" containerID="b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"
Sep 17 17:53:40.726955 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.726873    2581 scope.go:117] "RemoveContainer" containerID="fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"
Sep 17 17:53:40.727186 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:40.727167    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178\": container with ID starting with fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178 not found: ID does not exist" containerID="fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"
Sep 17 17:53:40.727229 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.727198    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178"} err="failed to get container status \"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178\": rpc error: code = NotFound desc = could not find container \"fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178\": container with ID starting with fd93f7c0c9dd6cac966a7265320823f8de6d8ff94df3bbb92dbf3d552eb3b178 not found: ID does not exist"
Sep 17 17:53:40.727229 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.727218    2581 scope.go:117] "RemoveContainer" containerID="7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"
Sep 17 17:53:40.727462 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:40.727443    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425\": container with ID starting with 7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425 not found: ID does not exist" containerID="7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"
Sep 17 17:53:40.727538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.727471    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425"} err="failed to get container status \"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425\": rpc error: code = NotFound desc = could not find container \"7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425\": container with ID starting with 7ba0dfff2a713e0e814d597bf25f750557e82cfec51cbe033905ee72744f7425 not found: ID does not exist"
Sep 17 17:53:40.727538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.727492    2581 scope.go:117] "RemoveContainer" containerID="b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"
Sep 17 17:53:40.727813 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:40.727793    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d\": container with ID starting with b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d not found: ID does not exist" containerID="b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"
Sep 17 17:53:40.727887 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.727818    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d"} err="failed to get container status \"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d\": rpc error: code = NotFound desc = could not find container \"b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d\": container with ID starting with b1b1fe487747ba3c7280dbb01f9415c034945b356ce30236883c59bee2ceb81d not found: ID does not exist"
Sep 17 17:53:40.729057 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.728959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:53:40.734408 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.734384    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-r5clr"]
Sep 17 17:53:40.786607 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.786547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:40.993857 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:40.993831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:53:40.995617 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:53:40.995584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bfad70_f782_49a0_b000_f4d4441689ea.slice/crio-543d0cd6d6b471e580249a3cf791352f0a06818ee0b0ae0a8662643e6f7f7ba5 WatchSource:0}: Error finding container 543d0cd6d6b471e580249a3cf791352f0a06818ee0b0ae0a8662643e6f7f7ba5: Status 404 returned error can't find the container with id 543d0cd6d6b471e580249a3cf791352f0a06818ee0b0ae0a8662643e6f7f7ba5
Sep 17 17:53:41.021979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.021953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:41.024258 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.024230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:41.253734 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.253633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:41.361849 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.361814    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08bff906-999f-4efc-b831-26c77b30b7ec" path="/var/lib/kubelet/pods/08bff906-999f-4efc-b831-26c77b30b7ec/volumes"
Sep 17 17:53:41.534597 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.534569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:53:41.535682 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:53:41.535654    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d40d9d7_30e9_4745_a158_c4c07d80e51d.slice/crio-a1d98a78b77cee973c40bd898457c218b12fff294fd3bf16e2658475ad6c0d15 WatchSource:0}: Error finding container a1d98a78b77cee973c40bd898457c218b12fff294fd3bf16e2658475ad6c0d15: Status 404 returned error can't find the container with id a1d98a78b77cee973c40bd898457c218b12fff294fd3bf16e2658475ad6c0d15
Sep 17 17:53:41.709422 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.709380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerStarted","Data":"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"}
Sep 17 17:53:41.709621 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.709454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerStarted","Data":"a1d98a78b77cee973c40bd898457c218b12fff294fd3bf16e2658475ad6c0d15"}
Sep 17 17:53:41.711044 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.711014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerStarted","Data":"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"}
Sep 17 17:53:41.711178 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:41.711069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerStarted","Data":"543d0cd6d6b471e580249a3cf791352f0a06818ee0b0ae0a8662643e6f7f7ba5"}
Sep 17 17:53:45.732002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:45.731970    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerID="279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552" exitCode=0
Sep 17 17:53:45.732421 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:45.732047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerDied","Data":"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"}
Sep 17 17:53:45.733600 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:45.733577    2581 generic.go:356] "Generic (PLEG): container finished" podID="45bfad70-f782-49a0-b000-f4d4441689ea" containerID="d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d" exitCode=0
Sep 17 17:53:45.733722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:45.733656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerDied","Data":"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"}
Sep 17 17:53:46.740336 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.740293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerStarted","Data":"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"}
Sep 17 17:53:46.740871 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.740395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerStarted","Data":"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"}
Sep 17 17:53:46.740871 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.740662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:46.742381 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.742360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerStarted","Data":"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"}
Sep 17 17:53:46.742465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.742402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerStarted","Data":"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"}
Sep 17 17:53:46.742671 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.742654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:46.742714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.742695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:46.744085 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.744065    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:53:46.775338 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.775284    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podStartSLOduration=3.189646007 podStartE2EDuration="6.775267359s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585621352s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:46.773478367 +0000 UTC m=+765.641611379" watchObservedRunningTime="2026-09-17 17:53:46.775267359 +0000 UTC m=+765.643400322"
Sep 17 17:53:46.794624 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:46.794575    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podStartSLOduration=3.138351441 podStartE2EDuration="6.794558665s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.656207224s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:46.793879592 +0000 UTC m=+765.662012555" watchObservedRunningTime="2026-09-17 17:53:46.794558665 +0000 UTC m=+765.662691628"
Sep 17 17:53:47.747142 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:47.747102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:47.747142 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:47.747118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:53:47.748393 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:47.748367    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:53:48.752268 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:48.752228    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:53:50.205184 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.205147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:50.355123 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.355085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:50.465591 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.465505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:53:50.465771 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.465747    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" containerID="cri-o://fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc" gracePeriod=30
Sep 17 17:53:50.465827 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.465795    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kube-rbac-proxy" containerID="cri-o://cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7" gracePeriod=30
Sep 17 17:53:50.515471 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.515419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:50.595613 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.595568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:50.674692 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.674628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:50.763465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.763367    2581 generic.go:356] "Generic (PLEG): container finished" podID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerID="cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7" exitCode=2
Sep 17 17:53:50.763465 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:50.763441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerDied","Data":"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"}
Sep 17 17:53:51.041930 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:51.041898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:51.322075 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:51.321993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:52.751786 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:52.751756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:53:52.752446 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:52.752408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:53:53.394300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:53.394266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 17:53:53.399609 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:53.399587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:53:53.757507 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:53.757429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:53:53.758122 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:53.758093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:53:55.412544 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.412520    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:53:55.553741 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.553710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config\") pod \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") "
Sep 17 17:53:55.553890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.553748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location\") pod \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") "
Sep 17 17:53:55.553890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.553781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkmnz\" (UniqueName: \"kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz\") pod \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") "
Sep 17 17:53:55.553890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.553798    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert\") pod \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") "
Sep 17 17:53:55.553890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.553828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls\") pod \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\" (UID: \"eee8a02f-0c12-4a6e-bb53-df9a11964b06\") "
Sep 17 17:53:55.554160 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.554129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eee8a02f-0c12-4a6e-bb53-df9a11964b06" (UID: "eee8a02f-0c12-4a6e-bb53-df9a11964b06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:55.554212 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.554164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eee8a02f-0c12-4a6e-bb53-df9a11964b06" (UID: "eee8a02f-0c12-4a6e-bb53-df9a11964b06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:55.554212 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.554186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eee8a02f-0c12-4a6e-bb53-df9a11964b06" (UID: "eee8a02f-0c12-4a6e-bb53-df9a11964b06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:55.555915 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.555886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz" (OuterVolumeSpecName: "kube-api-access-hkmnz") pod "eee8a02f-0c12-4a6e-bb53-df9a11964b06" (UID: "eee8a02f-0c12-4a6e-bb53-df9a11964b06"). InnerVolumeSpecName "kube-api-access-hkmnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:55.556280 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.556255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eee8a02f-0c12-4a6e-bb53-df9a11964b06" (UID: "eee8a02f-0c12-4a6e-bb53-df9a11964b06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:55.654591 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.654564    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.654591 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.654586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmnz\" (UniqueName: \"kubernetes.io/projected/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-api-access-hkmnz\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.654591 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.654595    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.654804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.654603    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eee8a02f-0c12-4a6e-bb53-df9a11964b06-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.654804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.654612    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eee8a02f-0c12-4a6e-bb53-df9a11964b06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.787965 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.787927    2581 generic.go:356] "Generic (PLEG): container finished" podID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerID="fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc" exitCode=0
Sep 17 17:53:55.788132 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.787994    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"
Sep 17 17:53:55.788132 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.788010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerDied","Data":"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"}
Sep 17 17:53:55.788132 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.788076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s" event={"ID":"eee8a02f-0c12-4a6e-bb53-df9a11964b06","Type":"ContainerDied","Data":"f2dfdfc65aeea203ce6629b25a6f0ea6e6c46d60fb7605289db25cf98d5849da"}
Sep 17 17:53:55.788132 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.788103    2581 scope.go:117] "RemoveContainer" containerID="cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"
Sep 17 17:53:55.796812 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.796797    2581 scope.go:117] "RemoveContainer" containerID="fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"
Sep 17 17:53:55.803996 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.803980    2581 scope.go:117] "RemoveContainer" containerID="8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"
Sep 17 17:53:55.813271 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.813253    2581 scope.go:117] "RemoveContainer" containerID="cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"
Sep 17 17:53:55.813590 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:55.813570    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7\": container with ID starting with cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7 not found: ID does not exist" containerID="cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"
Sep 17 17:53:55.813714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.813598    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7"} err="failed to get container status \"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7\": rpc error: code = NotFound desc = could not find container \"cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7\": container with ID starting with cd25480bc148e365f846757d52513822e68daae2d782703a797080ea7e0d57e7 not found: ID does not exist"
Sep 17 17:53:55.813714 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.813615    2581 scope.go:117] "RemoveContainer" containerID="fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"
Sep 17 17:53:55.813934 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:55.813918    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc\": container with ID starting with fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc not found: ID does not exist" containerID="fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"
Sep 17 17:53:55.813979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.813938    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc"} err="failed to get container status \"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc\": rpc error: code = NotFound desc = could not find container \"fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc\": container with ID starting with fe2e54594d8f07c05b3f1010104e6d67699d680da1780fb1dc35e142f78c46cc not found: ID does not exist"
Sep 17 17:53:55.813979 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.813952    2581 scope.go:117] "RemoveContainer" containerID="8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"
Sep 17 17:53:55.814281 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:53:55.814265    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4\": container with ID starting with 8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4 not found: ID does not exist" containerID="8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"
Sep 17 17:53:55.814331 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.814286    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4"} err="failed to get container status \"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4\": rpc error: code = NotFound desc = could not find container \"8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4\": container with ID starting with 8cb08b1b6fea914069159f803bda9bc634b50e98407746a9644604a091b534c4 not found: ID does not exist"
Sep 17 17:53:55.814516 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.814499    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:53:55.817248 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:55.817226    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-bbc14-predictor-6f6986cdb9-ddq7s"]
Sep 17 17:53:57.362011 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:53:57.361972    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" path="/var/lib/kubelet/pods/eee8a02f-0c12-4a6e-bb53-df9a11964b06/volumes"
Sep 17 17:54:02.753164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:02.753124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:03.758987 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:03.758899    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:54:12.753325 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:12.753284    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:13.758027 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:13.757977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:54:22.752944 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:22.752851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:23.758255 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:23.758214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:54:32.752820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:32.752782    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:33.758342 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:33.758284    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:54:42.753149 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:42.753107    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:43.758012 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:43.757974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:54:52.752445 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:52.752397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:54:53.758751 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:54:53.758705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:55:02.752891 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:02.752854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:55:03.759362 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:03.759335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:55:10.279777 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.279744    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:55:10.280200 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.279987    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" containerID="cri-o://7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9" gracePeriod=30
Sep 17 17:55:10.280200 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.280027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kube-rbac-proxy" containerID="cri-o://04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b" gracePeriod=30
Sep 17 17:55:10.299339 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.299315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:10.325919 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.325897    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:55:10.326169 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.326137    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" containerID="cri-o://c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43" gracePeriod=30
Sep 17 17:55:10.326169 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.326155    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kube-rbac-proxy" containerID="cri-o://75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7" gracePeriod=30
Sep 17 17:55:10.383749 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:10.383723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:11.094303 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:11.094267    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerID="04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b" exitCode=2
Sep 17 17:55:11.094476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:11.094335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerDied","Data":"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"}
Sep 17 17:55:11.096381 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:11.096362    2581 generic.go:356] "Generic (PLEG): container finished" podID="45bfad70-f782-49a0-b000-f4d4441689ea" containerID="75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7" exitCode=2
Sep 17 17:55:11.096494 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:11.096399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerDied","Data":"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"}
Sep 17 17:55:12.748147 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:12.748101    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 17:55:12.752752 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:12.752724    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:55:13.752408 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:13.752367    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 17:55:13.758078 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:13.758039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:55:14.998709 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:14.998686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:55:15.027614 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.027589    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:55:15.030139 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls\") pod \"45bfad70-f782-49a0-b000-f4d4441689ea\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") "
Sep 17 17:55:15.030268 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8ljs\" (UniqueName: \"kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs\") pod \"45bfad70-f782-49a0-b000-f4d4441689ea\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") "
Sep 17 17:55:15.030268 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030179    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config\") pod \"45bfad70-f782-49a0-b000-f4d4441689ea\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") "
Sep 17 17:55:15.030268 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert\") pod \"45bfad70-f782-49a0-b000-f4d4441689ea\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") "
Sep 17 17:55:15.030441 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location\") pod \"45bfad70-f782-49a0-b000-f4d4441689ea\" (UID: \"45bfad70-f782-49a0-b000-f4d4441689ea\") "
Sep 17 17:55:15.030590 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45bfad70-f782-49a0-b000-f4d4441689ea" (UID: "45bfad70-f782-49a0-b000-f4d4441689ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:15.030590 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45bfad70-f782-49a0-b000-f4d4441689ea" (UID: "45bfad70-f782-49a0-b000-f4d4441689ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:15.030862 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030761    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.030862 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030784    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45bfad70-f782-49a0-b000-f4d4441689ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.030862 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.030848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45bfad70-f782-49a0-b000-f4d4441689ea" (UID: "45bfad70-f782-49a0-b000-f4d4441689ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:15.032573 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.032550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45bfad70-f782-49a0-b000-f4d4441689ea" (UID: "45bfad70-f782-49a0-b000-f4d4441689ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:15.032741 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.032711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs" (OuterVolumeSpecName: "kube-api-access-b8ljs") pod "45bfad70-f782-49a0-b000-f4d4441689ea" (UID: "45bfad70-f782-49a0-b000-f4d4441689ea"). InnerVolumeSpecName "kube-api-access-b8ljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:15.115527 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.115495    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerID="7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9" exitCode=0
Sep 17 17:55:15.115736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.115546    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"
Sep 17 17:55:15.115736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.115582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerDied","Data":"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"}
Sep 17 17:55:15.115736 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.115703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr" event={"ID":"3d40d9d7-30e9-4745-a158-c4c07d80e51d","Type":"ContainerDied","Data":"a1d98a78b77cee973c40bd898457c218b12fff294fd3bf16e2658475ad6c0d15"}
Sep 17 17:55:15.115897 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.115752    2581 scope.go:117] "RemoveContainer" containerID="04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"
Sep 17 17:55:15.117679 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.117631    2581 generic.go:356] "Generic (PLEG): container finished" podID="45bfad70-f782-49a0-b000-f4d4441689ea" containerID="c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43" exitCode=0
Sep 17 17:55:15.117795 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.117703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"
Sep 17 17:55:15.117795 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.117732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerDied","Data":"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"}
Sep 17 17:55:15.117795 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.117777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk" event={"ID":"45bfad70-f782-49a0-b000-f4d4441689ea","Type":"ContainerDied","Data":"543d0cd6d6b471e580249a3cf791352f0a06818ee0b0ae0a8662643e6f7f7ba5"}
Sep 17 17:55:15.126210 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.125502    2581 scope.go:117] "RemoveContainer" containerID="7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"
Sep 17 17:55:15.131190 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szz2t\" (UniqueName: \"kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t\") pod \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") "
Sep 17 17:55:15.131284 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config\") pod \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") "
Sep 17 17:55:15.131284 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") pod \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") "
Sep 17 17:55:15.131284 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert\") pod \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") "
Sep 17 17:55:15.131450 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131287    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location\") pod \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\" (UID: \"3d40d9d7-30e9-4745-a158-c4c07d80e51d\") "
Sep 17 17:55:15.131506 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131486    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45bfad70-f782-49a0-b000-f4d4441689ea-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.131559 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131503    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ljs\" (UniqueName: \"kubernetes.io/projected/45bfad70-f782-49a0-b000-f4d4441689ea-kube-api-access-b8ljs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.131559 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131518    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45bfad70-f782-49a0-b000-f4d4441689ea-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.131683 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d40d9d7-30e9-4745-a158-c4c07d80e51d" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:15.131683 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d40d9d7-30e9-4745-a158-c4c07d80e51d" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:15.131808 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.131791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d40d9d7-30e9-4745-a158-c4c07d80e51d" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:15.133234 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.133215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t" (OuterVolumeSpecName: "kube-api-access-szz2t") pod "3d40d9d7-30e9-4745-a158-c4c07d80e51d" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d"). InnerVolumeSpecName "kube-api-access-szz2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:15.133335 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.133243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d40d9d7-30e9-4745-a158-c4c07d80e51d" (UID: "3d40d9d7-30e9-4745-a158-c4c07d80e51d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:15.134045 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.134031    2581 scope.go:117] "RemoveContainer" containerID="279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"
Sep 17 17:55:15.141101 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141080    2581 scope.go:117] "RemoveContainer" containerID="04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"
Sep 17 17:55:15.141386 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.141363    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b\": container with ID starting with 04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b not found: ID does not exist" containerID="04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"
Sep 17 17:55:15.141458 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141392    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b"} err="failed to get container status \"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b\": rpc error: code = NotFound desc = could not find container \"04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b\": container with ID starting with 04f4e040e5edf7a70bca6b72d2ba8bcb7236eb249f0a463254462de1767dd22b not found: ID does not exist"
Sep 17 17:55:15.141458 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141413    2581 scope.go:117] "RemoveContainer" containerID="7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"
Sep 17 17:55:15.141711 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.141695    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9\": container with ID starting with 7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9 not found: ID does not exist" containerID="7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"
Sep 17 17:55:15.141772 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141719    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9"} err="failed to get container status \"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9\": rpc error: code = NotFound desc = could not find container \"7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9\": container with ID starting with 7ac227735753a5a679d6c85554022961d641c81263b925ecdb31ed5da01d99a9 not found: ID does not exist"
Sep 17 17:55:15.141772 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141734    2581 scope.go:117] "RemoveContainer" containerID="279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"
Sep 17 17:55:15.141986 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.141972    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552\": container with ID starting with 279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552 not found: ID does not exist" containerID="279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"
Sep 17 17:55:15.142025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.141990    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552"} err="failed to get container status \"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552\": rpc error: code = NotFound desc = could not find container \"279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552\": container with ID starting with 279839614e815f11420149592a6225719dbe717cf69938faaa60e4d6880fb552 not found: ID does not exist"
Sep 17 17:55:15.142025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.142001    2581 scope.go:117] "RemoveContainer" containerID="75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"
Sep 17 17:55:15.143104 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.143083    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:55:15.147115 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.147096    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-z9bmk"]
Sep 17 17:55:15.149099 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.149086    2581 scope.go:117] "RemoveContainer" containerID="c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"
Sep 17 17:55:15.155897 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.155880    2581 scope.go:117] "RemoveContainer" containerID="d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"
Sep 17 17:55:15.162485 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.162469    2581 scope.go:117] "RemoveContainer" containerID="75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"
Sep 17 17:55:15.162722 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.162701    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7\": container with ID starting with 75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7 not found: ID does not exist" containerID="75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"
Sep 17 17:55:15.162791 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.162726    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7"} err="failed to get container status \"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7\": rpc error: code = NotFound desc = could not find container \"75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7\": container with ID starting with 75af9286ac2936d670023034aa46b39217bfb5a0b5b78e0ae1987472de03b7d7 not found: ID does not exist"
Sep 17 17:55:15.162791 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.162742    2581 scope.go:117] "RemoveContainer" containerID="c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"
Sep 17 17:55:15.162986 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.162970    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43\": container with ID starting with c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43 not found: ID does not exist" containerID="c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"
Sep 17 17:55:15.163022 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.162993    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43"} err="failed to get container status \"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43\": rpc error: code = NotFound desc = could not find container \"c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43\": container with ID starting with c1fb88765e40261be337d11bba3fce31e1fbb8601fbd255d4875784ea9b8ec43 not found: ID does not exist"
Sep 17 17:55:15.163022 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.163006    2581 scope.go:117] "RemoveContainer" containerID="d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"
Sep 17 17:55:15.163225 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:15.163207    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d\": container with ID starting with d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d not found: ID does not exist" containerID="d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"
Sep 17 17:55:15.163262 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.163230    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d"} err="failed to get container status \"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d\": rpc error: code = NotFound desc = could not find container \"d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d\": container with ID starting with d7aad9138618d95cea59fdff058b5a5216cca2ba1f63c54fa2d3246c833b162d not found: ID does not exist"
Sep 17 17:55:15.231840 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.231820    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.231840 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.231838    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szz2t\" (UniqueName: \"kubernetes.io/projected/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-api-access-szz2t\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.231960 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.231848    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.231960 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.231856    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d40d9d7-30e9-4745-a158-c4c07d80e51d-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.231960 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.231864    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d9d7-30e9-4745-a158-c4c07d80e51d-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:55:15.361246 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.361221    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" path="/var/lib/kubelet/pods/45bfad70-f782-49a0-b000-f4d4441689ea/volumes"
Sep 17 17:55:15.448526 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.448492    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:55:15.452160 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:15.452141    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-mkqnr"]
Sep 17 17:55:17.361479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:17.361446    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" path="/var/lib/kubelet/pods/3d40d9d7-30e9-4745-a158-c4c07d80e51d/volumes"
Sep 17 17:55:20.517875 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.517837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:20.554801 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.554765    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:55:20.555077 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555063    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555081    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555095    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555112    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555118    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="storage-initializer"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555132    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="storage-initializer"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555139    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555144    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555151    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="storage-initializer"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555157    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="storage-initializer"
Sep 17 17:55:20.555164 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555181    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555186    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555191    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="storage-initializer"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="storage-initializer"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555247    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kserve-container"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kserve-container"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555263    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555269    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40d9d7-30e9-4745-a158-c4c07d80e51d" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555276    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bfad70-f782-49a0-b000-f4d4441689ea" containerName="kube-rbac-proxy"
Sep 17 17:55:20.555565 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.555281    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee8a02f-0c12-4a6e-bb53-df9a11964b06" containerName="kserve-container"
Sep 17 17:55:20.558169 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.558154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.561558 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.561536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5a8a9a-kube-rbac-proxy-sar-config\""
Sep 17 17:55:20.561917 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.561618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:55:20.561917 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.561623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5a8a9a-predictor-serving-cert\""
Sep 17 17:55:20.561917 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.561625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:55:20.563075 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.563060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:55:20.563143 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.563111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:55:20.563204 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.563152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:55:20.570797 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.570773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkg5\" (UniqueName: \"kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.570950 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.570813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.570950 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.570834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.570950 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.570849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.571146 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.570947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.573685 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.573666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:55:20.671479 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.671732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkg5\" (UniqueName: \"kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.671732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.671732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.671732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.671732 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:20.671705    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-serving-cert: secret "isvc-primary-5a8a9a-predictor-serving-cert" not found
Sep 17 17:55:20.672032 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:20.671786    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls podName:b58936e1-41a5-4674-ac58-b1dcf02e0cdf nodeName:}" failed. No retries permitted until 2026-09-17 17:55:21.171765307 +0000 UTC m=+860.039898261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls") pod "isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf") : secret "isvc-primary-5a8a9a-predictor-serving-cert" not found
Sep 17 17:55:20.672032 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.671933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.672113 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.672093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.672184 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.672167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.681203 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.681175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkg5\" (UniqueName: \"kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:20.870228 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:20.870191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:21.176899 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:21.176816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:21.179156 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:21.179138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") pod \"isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") " pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:21.467961 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:21.467872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:21.738661 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:21.738615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:55:21.740564 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:55:21.740535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58936e1_41a5_4674_ac58_b1dcf02e0cdf.slice/crio-b070400d4a10aac616a2590686d06a4828ee16ee29a8e8961f217b0e33d29ba8 WatchSource:0}: Error finding container b070400d4a10aac616a2590686d06a4828ee16ee29a8e8961f217b0e33d29ba8: Status 404 returned error can't find the container with id b070400d4a10aac616a2590686d06a4828ee16ee29a8e8961f217b0e33d29ba8
Sep 17 17:55:22.154343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:22.154305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerStarted","Data":"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"}
Sep 17 17:55:22.154538 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:22.154369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerStarted","Data":"b070400d4a10aac616a2590686d06a4828ee16ee29a8e8961f217b0e33d29ba8"}
Sep 17 17:55:26.173360 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:26.173325    2581 generic.go:356] "Generic (PLEG): container finished" podID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerID="c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5" exitCode=0
Sep 17 17:55:26.173758 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:26.173390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerDied","Data":"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"}
Sep 17 17:55:27.179686 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.179632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerStarted","Data":"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"}
Sep 17 17:55:27.180067 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.179715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerStarted","Data":"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"}
Sep 17 17:55:27.180067 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.179970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:27.180067 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.180015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:27.181447 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.181421    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:55:27.213282 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:27.213234    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podStartSLOduration=3.381978036 podStartE2EDuration="7.213220853s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831242817s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:27.211531714 +0000 UTC m=+866.079664698" watchObservedRunningTime="2026-09-17 17:55:27.213220853 +0000 UTC m=+866.081353827"
Sep 17 17:55:28.184149 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:28.184110    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:55:30.443546 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:30.443506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:30.664611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:30.664571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:33.189306 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:33.189275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:55:33.189824 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:33.189794    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:55:40.535849 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.535796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:40.639353 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.639321    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:40.672234 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.672208    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:55:40.675833 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.675815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.678802 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.678780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-39c63-predictor-serving-cert\""
Sep 17 17:55:40.679415 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.679393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-39c63-kube-rbac-proxy-sar-config\""
Sep 17 17:55:40.693599 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.693572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:55:40.737480 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.737455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sw9c\" (UniqueName: \"kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.737611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.737496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.737611 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.737586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.751165 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.751144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:40.838053 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.837969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sw9c\" (UniqueName: \"kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.838053 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.838013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.838278 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.838061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.838278 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:40.838158    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-serving-cert: secret "message-dumper-raw-39c63-predictor-serving-cert" not found
Sep 17 17:55:40.838278 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:40.838233    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls podName:3dbee895-fe94-4cba-bb83-521b79b34ab2 nodeName:}" failed. No retries permitted until 2026-09-17 17:55:41.338213852 +0000 UTC m=+880.206346794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls") pod "message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" (UID: "3dbee895-fe94-4cba-bb83-521b79b34ab2") : secret "message-dumper-raw-39c63-predictor-serving-cert" not found
Sep 17 17:55:40.838776 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.838751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.847188 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.847165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sw9c\" (UniqueName: \"kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:40.850206 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:40.850176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:41.342520 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:41.342488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:41.342728 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:41.342628    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-serving-cert: secret "message-dumper-raw-39c63-predictor-serving-cert" not found
Sep 17 17:55:41.342796 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:55:41.342730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls podName:3dbee895-fe94-4cba-bb83-521b79b34ab2 nodeName:}" failed. No retries permitted until 2026-09-17 17:55:42.342714504 +0000 UTC m=+881.210847450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls") pod "message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" (UID: "3dbee895-fe94-4cba-bb83-521b79b34ab2") : secret "message-dumper-raw-39c63-predictor-serving-cert" not found
Sep 17 17:55:41.660939 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:41.660848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:42.353007 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:42.352961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:42.355378 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:42.355350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"message-dumper-raw-39c63-predictor-66ff79587d-s6vgg\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") " pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:42.519632 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:42.519603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:42.792849 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:42.792824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:55:42.795597 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:55:42.795560    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbee895_fe94_4cba_bb83_521b79b34ab2.slice/crio-252a6a7debd18ea03a26be698c55c311eb843412f96d1c855cb22d8c7458a72a WatchSource:0}: Error finding container 252a6a7debd18ea03a26be698c55c311eb843412f96d1c855cb22d8c7458a72a: Status 404 returned error can't find the container with id 252a6a7debd18ea03a26be698c55c311eb843412f96d1c855cb22d8c7458a72a
Sep 17 17:55:42.797469 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:42.797453    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:55:43.190057 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:43.189972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:55:43.253102 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:43.253067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerStarted","Data":"252a6a7debd18ea03a26be698c55c311eb843412f96d1c855cb22d8c7458a72a"}
Sep 17 17:55:44.259916 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.259795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerStarted","Data":"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"}
Sep 17 17:55:44.259916 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.259909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:44.260324 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.259943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerStarted","Data":"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"}
Sep 17 17:55:44.260324 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.259974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:44.262215 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.262190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:44.282082 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:44.282037    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" podStartSLOduration=3.181545199 podStartE2EDuration="4.282024436s" totalImagesPullingTime="1.100479237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:44.280327829 +0000 UTC m=+883.148460812" watchObservedRunningTime="2026-09-17 17:55:44.282024436 +0000 UTC m=+883.150157399"
Sep 17 17:55:51.272463 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:51.272427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:55:53.190485 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:55:53.190446    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:00.810424 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:00.810383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:00.962300 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:00.962264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:03.190264 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:03.190221    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:03.292770 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:03.292742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:56:03.323149 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:03.323125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 17:56:07.223651 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:07.223613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:13.190710 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:13.190603    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:23.190177 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:23.190138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:33.190088 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:33.190044    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:35.356436 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:35.356390    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:56:45.361124 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:56:45.361092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:57:05.662346 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:05.662316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:07.044834 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:07.044797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:21.812000 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:21.811896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:21.823183 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:21.823150    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:57:21.823475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:21.823450    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" containerID="cri-o://dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4" gracePeriod=30
Sep 17 17:57:21.823578 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:21.823544    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kube-rbac-proxy" containerID="cri-o://bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3" gracePeriod=30
Sep 17 17:57:21.887371 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:21.887335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:22.651540 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:22.651507    2581 generic.go:356] "Generic (PLEG): container finished" podID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerID="bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3" exitCode=2
Sep 17 17:57:22.651732 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:22.651566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerDied","Data":"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"}
Sep 17 17:57:23.184894 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:23.184844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 17:57:25.356514 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.356471    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 17:57:25.720475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.720389    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-39c63-predictor-66ff79587d-s6vgg_3dbee895-fe94-4cba-bb83-521b79b34ab2/kserve-container/0.log"
Sep 17 17:57:25.963694 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.963655    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:25.980498 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.980432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:57:25.980705 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.980677    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kserve-container" containerID="cri-o://ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd" gracePeriod=30
Sep 17 17:57:25.980774 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:25.980694    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kube-rbac-proxy" containerID="cri-o://309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4" gracePeriod=30
Sep 17 17:57:26.066514 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.066476    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:26.153386 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.153353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:26.231220 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.231127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:26.281650 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.281627    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:57:26.284481 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.284465    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:57:26.446975 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.446951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sw9c\" (UniqueName: \"kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c\") pod \"3dbee895-fe94-4cba-bb83-521b79b34ab2\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.446985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location\") pod \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config\") pod \"3dbee895-fe94-4cba-bb83-521b79b34ab2\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert\") pod \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzkg5\" (UniqueName: \"kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5\") pod \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config\") pod \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") pod \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\" (UID: \"b58936e1-41a5-4674-ac58-b1dcf02e0cdf\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") pod \"3dbee895-fe94-4cba-bb83-521b79b34ab2\" (UID: \"3dbee895-fe94-4cba-bb83-521b79b34ab2\") "
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447398    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b58936e1-41a5-4674-ac58-b1dcf02e0cdf" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:26.447457 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b58936e1-41a5-4674-ac58-b1dcf02e0cdf" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:26.447918 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b58936e1-41a5-4674-ac58-b1dcf02e0cdf" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:26.447918 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.447722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3dbee895-fe94-4cba-bb83-521b79b34ab2" (UID: "3dbee895-fe94-4cba-bb83-521b79b34ab2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:26.449240 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.449213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b58936e1-41a5-4674-ac58-b1dcf02e0cdf" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:26.449378 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.449320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5" (OuterVolumeSpecName: "kube-api-access-pzkg5") pod "b58936e1-41a5-4674-ac58-b1dcf02e0cdf" (UID: "b58936e1-41a5-4674-ac58-b1dcf02e0cdf"). InnerVolumeSpecName "kube-api-access-pzkg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:26.449528 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.449508    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c" (OuterVolumeSpecName: "kube-api-access-5sw9c") pod "3dbee895-fe94-4cba-bb83-521b79b34ab2" (UID: "3dbee895-fe94-4cba-bb83-521b79b34ab2"). InnerVolumeSpecName "kube-api-access-5sw9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:26.449596 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.449582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dbee895-fe94-4cba-bb83-521b79b34ab2" (UID: "3dbee895-fe94-4cba-bb83-521b79b34ab2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:26.548336 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548311    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzkg5\" (UniqueName: \"kubernetes.io/projected/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-api-access-pzkg5\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548336 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548336    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548349    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548359    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dbee895-fe94-4cba-bb83-521b79b34ab2-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548370    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sw9c\" (UniqueName: \"kubernetes.io/projected/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-api-access-5sw9c\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548381    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548393    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dbee895-fe94-4cba-bb83-521b79b34ab2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.548522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.548408    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b58936e1-41a5-4674-ac58-b1dcf02e0cdf-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.668332 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.668299    2581 generic.go:356] "Generic (PLEG): container finished" podID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerID="dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4" exitCode=0
Sep 17 17:57:26.668467 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.668338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerDied","Data":"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"}
Sep 17 17:57:26.668467 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.668348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"
Sep 17 17:57:26.668467 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.668409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh" event={"ID":"b58936e1-41a5-4674-ac58-b1dcf02e0cdf","Type":"ContainerDied","Data":"b070400d4a10aac616a2590686d06a4828ee16ee29a8e8961f217b0e33d29ba8"}
Sep 17 17:57:26.668467 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.668466    2581 scope.go:117] "RemoveContainer" containerID="bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"
Sep 17 17:57:26.670053 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670027    2581 generic.go:356] "Generic (PLEG): container finished" podID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerID="309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4" exitCode=2
Sep 17 17:57:26.670053 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670054    2581 generic.go:356] "Generic (PLEG): container finished" podID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerID="ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd" exitCode=2
Sep 17 17:57:26.670213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670069    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"
Sep 17 17:57:26.670213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerDied","Data":"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"}
Sep 17 17:57:26.670213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerDied","Data":"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"}
Sep 17 17:57:26.670213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.670167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" event={"ID":"3dbee895-fe94-4cba-bb83-521b79b34ab2","Type":"ContainerDied","Data":"252a6a7debd18ea03a26be698c55c311eb843412f96d1c855cb22d8c7458a72a"}
Sep 17 17:57:26.676708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.676685    2581 scope.go:117] "RemoveContainer" containerID="dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"
Sep 17 17:57:26.684129 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.684111    2581 scope.go:117] "RemoveContainer" containerID="c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"
Sep 17 17:57:26.690908 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.690893    2581 scope.go:117] "RemoveContainer" containerID="bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"
Sep 17 17:57:26.691165 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:26.691148    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3\": container with ID starting with bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3 not found: ID does not exist" containerID="bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"
Sep 17 17:57:26.691213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691174    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3"} err="failed to get container status \"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3\": rpc error: code = NotFound desc = could not find container \"bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3\": container with ID starting with bbf66251e0f1b9e85e62f65ac10b617ab96bf2e71da16b32e70fad32c6b7a6b3 not found: ID does not exist"
Sep 17 17:57:26.691213 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691189    2581 scope.go:117] "RemoveContainer" containerID="dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"
Sep 17 17:57:26.691442 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:26.691425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4\": container with ID starting with dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4 not found: ID does not exist" containerID="dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"
Sep 17 17:57:26.691476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691449    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4"} err="failed to get container status \"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4\": rpc error: code = NotFound desc = could not find container \"dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4\": container with ID starting with dd549bfbf531bec12a0c659918e9b3f5f817530bed6bb3b8692115d682c7a5a4 not found: ID does not exist"
Sep 17 17:57:26.691476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691462    2581 scope.go:117] "RemoveContainer" containerID="c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"
Sep 17 17:57:26.691676 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:26.691656    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5\": container with ID starting with c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5 not found: ID does not exist" containerID="c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"
Sep 17 17:57:26.691726 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5"} err="failed to get container status \"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5\": rpc error: code = NotFound desc = could not find container \"c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5\": container with ID starting with c0b764c5e014e4cba274db362b350007193f678b248cb33d0c39f0829711a8b5 not found: ID does not exist"
Sep 17 17:57:26.691726 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.691697    2581 scope.go:117] "RemoveContainer" containerID="309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"
Sep 17 17:57:26.698292 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.698274    2581 scope.go:117] "RemoveContainer" containerID="ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"
Sep 17 17:57:26.704830 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.704814    2581 scope.go:117] "RemoveContainer" containerID="309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"
Sep 17 17:57:26.705066 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:26.705047    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4\": container with ID starting with 309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4 not found: ID does not exist" containerID="309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"
Sep 17 17:57:26.705125 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"} err="failed to get container status \"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4\": rpc error: code = NotFound desc = could not find container \"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4\": container with ID starting with 309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4 not found: ID does not exist"
Sep 17 17:57:26.705125 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705086    2581 scope.go:117] "RemoveContainer" containerID="ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"
Sep 17 17:57:26.705331 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:26.705315    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd\": container with ID starting with ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd not found: ID does not exist" containerID="ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"
Sep 17 17:57:26.705379 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705335    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"} err="failed to get container status \"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd\": rpc error: code = NotFound desc = could not find container \"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd\": container with ID starting with ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd not found: ID does not exist"
Sep 17 17:57:26.705379 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705349    2581 scope.go:117] "RemoveContainer" containerID="309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"
Sep 17 17:57:26.705545 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705529    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4"} err="failed to get container status \"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4\": rpc error: code = NotFound desc = could not find container \"309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4\": container with ID starting with 309e7ff7211c8d8b34782db6deeb5c8c2271abdc5578cab8e5cfa668d06628f4 not found: ID does not exist"
Sep 17 17:57:26.705587 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705545    2581 scope.go:117] "RemoveContainer" containerID="ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"
Sep 17 17:57:26.705848 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.705831    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd"} err="failed to get container status \"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd\": rpc error: code = NotFound desc = could not find container \"ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd\": container with ID starting with ffbeed6bad272dc7194c745917f3a8a536ea86a86b402787e10f361f11c2d7dd not found: ID does not exist"
Sep 17 17:57:26.721525 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.721507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:57:26.727840 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.727789    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg"]
Sep 17 17:57:26.757726 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.757704    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:57:26.759024 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:26.759004    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5a8a9a-predictor-5b79f85564-tjmfh"]
Sep 17 17:57:27.029475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.029442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.125030 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.124994    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:27.125419 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125400    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="storage-initializer"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="storage-initializer"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125434    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125442    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125460    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kube-rbac-proxy"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125469    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kube-rbac-proxy"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125484    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kube-rbac-proxy"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kube-rbac-proxy"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125501    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kserve-container"
Sep 17 17:57:27.125513 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125509    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kserve-container"
Sep 17 17:57:27.125974 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125593    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kube-rbac-proxy"
Sep 17 17:57:27.125974 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125606    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" containerName="kserve-container"
Sep 17 17:57:27.125974 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125617    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kserve-container"
Sep 17 17:57:27.125974 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.125624    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kube-rbac-proxy"
Sep 17 17:57:27.130399 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.130380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.133727 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.133694    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a280a8-predictor-serving-cert\""
Sep 17 17:57:27.133727 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.133721    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:27.133890 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.133775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a280a8\""
Sep 17 17:57:27.134000 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.133983    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:27.134066 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.133997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a280a8-dockercfg-hnpsf\""
Sep 17 17:57:27.134066 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.134011    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a280a8-kube-rbac-proxy-sar-config\""
Sep 17 17:57:27.134152 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.134066    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:27.141140 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.141119    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:27.253626 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.253542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.253626 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.253616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmgs\" (UniqueName: \"kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.253820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.253668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.253820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.253696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.253820 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.253738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.268428 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.268396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-39c63-predictor-66ff79587d-s6vgg" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:57:27.354233 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmgs\" (UniqueName: \"kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354233 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354410 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354471 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354471 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354578 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:27.354491    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-serving-cert: secret "isvc-init-fail-a280a8-predictor-serving-cert" not found
Sep 17 17:57:27.354578 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:27.354541    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls podName:fc250708-7a66-4a06-9ad2-4c526cf51787 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:27.854525676 +0000 UTC m=+986.722658632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls") pod "isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787") : secret "isvc-init-fail-a280a8-predictor-serving-cert" not found
Sep 17 17:57:27.354728 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354917 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.354969 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.354953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.361787 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.361762    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dbee895-fe94-4cba-bb83-521b79b34ab2" path="/var/lib/kubelet/pods/3dbee895-fe94-4cba-bb83-521b79b34ab2/volumes"
Sep 17 17:57:27.362372 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.362353    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58936e1-41a5-4674-ac58-b1dcf02e0cdf" path="/var/lib/kubelet/pods/b58936e1-41a5-4674-ac58-b1dcf02e0cdf/volumes"
Sep 17 17:57:27.364131 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.364111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmgs\" (UniqueName: \"kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.483762 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.483732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.858237 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.858201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:27.860709 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:27.860688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") pod \"isvc-init-fail-a280a8-predictor-7598f757d9-lmggh\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") " pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:28.040340 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:28.040302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:28.313260 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:28.313236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:28.315562 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:57:28.315530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc250708_7a66_4a06_9ad2_4c526cf51787.slice/crio-16a0812a48fd74efbe5bed17020dc18325f8096e26da21c962323277d0d93de1 WatchSource:0}: Error finding container 16a0812a48fd74efbe5bed17020dc18325f8096e26da21c962323277d0d93de1: Status 404 returned error can't find the container with id 16a0812a48fd74efbe5bed17020dc18325f8096e26da21c962323277d0d93de1
Sep 17 17:57:28.681719 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:28.681623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerStarted","Data":"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"}
Sep 17 17:57:28.682070 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:28.681716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerStarted","Data":"16a0812a48fd74efbe5bed17020dc18325f8096e26da21c962323277d0d93de1"}
Sep 17 17:57:34.706780 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:34.706752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/0.log"
Sep 17 17:57:34.707151 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:34.706788    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerID="19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf" exitCode=1
Sep 17 17:57:34.707151 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:34.706835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerDied","Data":"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"}
Sep 17 17:57:35.712092 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:35.712065    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/0.log"
Sep 17 17:57:35.712440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:35.712188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerStarted","Data":"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"}
Sep 17 17:57:36.947688 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:36.947658    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:36.948188 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:36.947861    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer" containerID="cri-o://df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081" gracePeriod=30
Sep 17 17:57:36.976434 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:36.976390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.890595 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.890575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/1.log"
Sep 17 17:57:38.890958 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.890944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/0.log"
Sep 17 17:57:38.893455 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.893441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:38.949494 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert\") pod \"fc250708-7a66-4a06-9ad2-4c526cf51787\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") "
Sep 17 17:57:38.949628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfmgs\" (UniqueName: \"kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs\") pod \"fc250708-7a66-4a06-9ad2-4c526cf51787\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") "
Sep 17 17:57:38.949628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config\") pod \"fc250708-7a66-4a06-9ad2-4c526cf51787\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") "
Sep 17 17:57:38.949628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") pod \"fc250708-7a66-4a06-9ad2-4c526cf51787\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") "
Sep 17 17:57:38.949628 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location\") pod \"fc250708-7a66-4a06-9ad2-4c526cf51787\" (UID: \"fc250708-7a66-4a06-9ad2-4c526cf51787\") "
Sep 17 17:57:38.949911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc250708-7a66-4a06-9ad2-4c526cf51787" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:38.950010 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949986    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc250708-7a66-4a06-9ad2-4c526cf51787" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:38.950010 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.949990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc250708-7a66-4a06-9ad2-4c526cf51787" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:38.951593 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.951572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs" (OuterVolumeSpecName: "kube-api-access-wfmgs") pod "fc250708-7a66-4a06-9ad2-4c526cf51787" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787"). InnerVolumeSpecName "kube-api-access-wfmgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:38.951699 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:38.951678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc250708-7a66-4a06-9ad2-4c526cf51787" (UID: "fc250708-7a66-4a06-9ad2-4c526cf51787"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:39.050752 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.050723    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmgs\" (UniqueName: \"kubernetes.io/projected/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-api-access-wfmgs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:39.050752 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.050749    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:39.050946 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.050760    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc250708-7a66-4a06-9ad2-4c526cf51787-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:39.050946 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.050770    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc250708-7a66-4a06-9ad2-4c526cf51787-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:39.050946 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.050779    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc250708-7a66-4a06-9ad2-4c526cf51787-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:57:39.728475 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/1.log"
Sep 17 17:57:39.728804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728790    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a280a8-predictor-7598f757d9-lmggh_fc250708-7a66-4a06-9ad2-4c526cf51787/storage-initializer/0.log"
Sep 17 17:57:39.728868 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728822    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerID="df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081" exitCode=1
Sep 17 17:57:39.728907 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728878    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"
Sep 17 17:57:39.728947 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerDied","Data":"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"}
Sep 17 17:57:39.728984 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.728968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh" event={"ID":"fc250708-7a66-4a06-9ad2-4c526cf51787","Type":"ContainerDied","Data":"16a0812a48fd74efbe5bed17020dc18325f8096e26da21c962323277d0d93de1"}
Sep 17 17:57:39.729025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.729006    2581 scope.go:117] "RemoveContainer" containerID="df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"
Sep 17 17:57:39.736957 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.736934    2581 scope.go:117] "RemoveContainer" containerID="19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"
Sep 17 17:57:39.744087 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.744070    2581 scope.go:117] "RemoveContainer" containerID="df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"
Sep 17 17:57:39.744343 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:39.744325    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081\": container with ID starting with df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081 not found: ID does not exist" containerID="df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"
Sep 17 17:57:39.744402 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.744349    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081"} err="failed to get container status \"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081\": rpc error: code = NotFound desc = could not find container \"df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081\": container with ID starting with df8230c37f74238c3c1677bd4fb3049cde90c655e29685c491e84b641f47d081 not found: ID does not exist"
Sep 17 17:57:39.744402 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.744364    2581 scope.go:117] "RemoveContainer" containerID="19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"
Sep 17 17:57:39.744632 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:57:39.744591    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf\": container with ID starting with 19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf not found: ID does not exist" containerID="19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"
Sep 17 17:57:39.744722 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.744659    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf"} err="failed to get container status \"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf\": rpc error: code = NotFound desc = could not find container \"19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf\": container with ID starting with 19c0e277c4f6af8e26c527b6f8921fc6457044364026953de6afc87fd9316ecf not found: ID does not exist"
Sep 17 17:57:39.796199 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.796175    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:39.809450 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:39.809421    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a280a8-predictor-7598f757d9-lmggh"]
Sep 17 17:57:41.361223 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:41.361187    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" path="/var/lib/kubelet/pods/fc250708-7a66-4a06-9ad2-4c526cf51787/volumes"
Sep 17 17:57:42.129273 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.129242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:42.168831 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.168804    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:57:42.169127 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.169178 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.169178 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.169178 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169151    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.169286 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169234    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.169286 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.169242    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc250708-7a66-4a06-9ad2-4c526cf51787" containerName="storage-initializer"
Sep 17 17:57:42.173626 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.173611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.176793 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.176772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:42.178181 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:57:42.178181 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-69571-kube-rbac-proxy-sar-config\""
Sep 17 17:57:42.178343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178226    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:42.178343 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-69571-predictor-serving-cert\""
Sep 17 17:57:42.178456 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178372    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:42.178515 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.178456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:42.182870 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.182848    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:57:42.224838 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.224813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:42.277569 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.277548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.277693 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.277580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhhn\" (UniqueName: \"kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.277693 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.277602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.277807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.277749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.277807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.277779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.378992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhhn\" (UniqueName: \"kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379025 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379440 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379573 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379814 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.379911 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.379848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.381673 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.381655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.388483 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.388463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhhn\" (UniqueName: \"kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn\") pod \"raw-sklearn-69571-predictor-7f9bb65cf7-wc74s\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") " pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.484070 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.484045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:42.757609 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:42.757581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:57:42.759493 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:57:42.759459    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4473f1a6_6449_4c10_97c0_68205f32cf85.slice/crio-b823dd3ab65c68e88cc31d693b5c70ce6a32dee2fae9d746e256c91be54f25ac WatchSource:0}: Error finding container b823dd3ab65c68e88cc31d693b5c70ce6a32dee2fae9d746e256c91be54f25ac: Status 404 returned error can't find the container with id b823dd3ab65c68e88cc31d693b5c70ce6a32dee2fae9d746e256c91be54f25ac
Sep 17 17:57:43.748997 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:43.748956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerStarted","Data":"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"}
Sep 17 17:57:43.749476 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:43.749026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerStarted","Data":"b823dd3ab65c68e88cc31d693b5c70ce6a32dee2fae9d746e256c91be54f25ac"}
Sep 17 17:57:46.763062 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:46.762974    2581 generic.go:356] "Generic (PLEG): container finished" podID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerID="971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1" exitCode=0
Sep 17 17:57:46.763487 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:46.763050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerDied","Data":"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"}
Sep 17 17:57:47.774370 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:47.774341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerStarted","Data":"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"}
Sep 17 17:57:47.774757 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:47.774428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerStarted","Data":"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"}
Sep 17 17:57:47.774757 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:47.774618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:47.818094 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:47.818052    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podStartSLOduration=2.309024566 podStartE2EDuration="5.818039068s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.509014502s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:47.817100538 +0000 UTC m=+1006.685233501" watchObservedRunningTime="2026-09-17 17:57:47.818039068 +0000 UTC m=+1006.686172032"
Sep 17 17:57:48.779072 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:48.779039    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:48.780588 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:48.780551    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:49.783492 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:49.783448    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:57:54.788772 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:54.788742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:57:54.789304 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:57:54.789278    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:04.789550 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:04.789510    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:14.789861 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:14.789823    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:24.789989 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:24.789944    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:34.790154 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:34.790115    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:44.789552 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:44.789474    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:58:54.789366 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:58:54.789317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:59:04.790637 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:04.790607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:59:12.301807 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:12.301773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:59:12.302195 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:12.302046    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" containerID="cri-o://433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24" gracePeriod=30
Sep 17 17:59:12.302195 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:12.302094    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kube-rbac-proxy" containerID="cri-o://cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6" gracePeriod=30
Sep 17 17:59:12.351085 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:12.351046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:13.111824 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:13.111785    2581 generic.go:356] "Generic (PLEG): container finished" podID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerID="cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6" exitCode=2
Sep 17 17:59:13.111988 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:13.111857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerDied","Data":"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"}
Sep 17 17:59:14.783708 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:14.783670    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:59:14.790044 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:14.790021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:59:16.852295 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.852270    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:59:16.968631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.968533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert\") pod \"4473f1a6-6449-4c10-97c0-68205f32cf85\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") "
Sep 17 17:59:16.968631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.968588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config\") pod \"4473f1a6-6449-4c10-97c0-68205f32cf85\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") "
Sep 17 17:59:16.968631 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.968633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location\") pod \"4473f1a6-6449-4c10-97c0-68205f32cf85\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") "
Sep 17 17:59:16.968943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.968704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lhhn\" (UniqueName: \"kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn\") pod \"4473f1a6-6449-4c10-97c0-68205f32cf85\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") "
Sep 17 17:59:16.968943 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.968722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls\") pod \"4473f1a6-6449-4c10-97c0-68205f32cf85\" (UID: \"4473f1a6-6449-4c10-97c0-68205f32cf85\") "
Sep 17 17:59:16.969057 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.969028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4473f1a6-6449-4c10-97c0-68205f32cf85" (UID: "4473f1a6-6449-4c10-97c0-68205f32cf85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:16.969116 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.969025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4473f1a6-6449-4c10-97c0-68205f32cf85" (UID: "4473f1a6-6449-4c10-97c0-68205f32cf85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:16.969116 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.969070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4473f1a6-6449-4c10-97c0-68205f32cf85" (UID: "4473f1a6-6449-4c10-97c0-68205f32cf85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:16.970876 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.970854    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn" (OuterVolumeSpecName: "kube-api-access-9lhhn") pod "4473f1a6-6449-4c10-97c0-68205f32cf85" (UID: "4473f1a6-6449-4c10-97c0-68205f32cf85"). InnerVolumeSpecName "kube-api-access-9lhhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:16.970987 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:16.970968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4473f1a6-6449-4c10-97c0-68205f32cf85" (UID: "4473f1a6-6449-4c10-97c0-68205f32cf85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:17.069567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.069524    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4473f1a6-6449-4c10-97c0-68205f32cf85-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:59:17.069567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.069561    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhhn\" (UniqueName: \"kubernetes.io/projected/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-api-access-9lhhn\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:59:17.069567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.069571    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4473f1a6-6449-4c10-97c0-68205f32cf85-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:59:17.069567 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.069581    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:59:17.069815 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.069589    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4473f1a6-6449-4c10-97c0-68205f32cf85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 17:59:17.130804 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.130777    2581 generic.go:356] "Generic (PLEG): container finished" podID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerID="433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24" exitCode=0
Sep 17 17:59:17.130933 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.130824    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"
Sep 17 17:59:17.130933 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.130854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerDied","Data":"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"}
Sep 17 17:59:17.130933 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.130913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s" event={"ID":"4473f1a6-6449-4c10-97c0-68205f32cf85","Type":"ContainerDied","Data":"b823dd3ab65c68e88cc31d693b5c70ce6a32dee2fae9d746e256c91be54f25ac"}
Sep 17 17:59:17.131062 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.130944    2581 scope.go:117] "RemoveContainer" containerID="cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"
Sep 17 17:59:17.139957 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.139937    2581 scope.go:117] "RemoveContainer" containerID="433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"
Sep 17 17:59:17.149923 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.149900    2581 scope.go:117] "RemoveContainer" containerID="971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"
Sep 17 17:59:17.157375 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.157355    2581 scope.go:117] "RemoveContainer" containerID="cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"
Sep 17 17:59:17.157624 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:59:17.157607    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6\": container with ID starting with cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6 not found: ID does not exist" containerID="cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"
Sep 17 17:59:17.157702 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.157629    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6"} err="failed to get container status \"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6\": rpc error: code = NotFound desc = could not find container \"cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6\": container with ID starting with cffec8434e90b03435cc9424ab828bfd9ed8c23d4d567d8aaddd50a755b9a4c6 not found: ID does not exist"
Sep 17 17:59:17.157702 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.157687    2581 scope.go:117] "RemoveContainer" containerID="433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"
Sep 17 17:59:17.157933 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:59:17.157916    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24\": container with ID starting with 433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24 not found: ID does not exist" containerID="433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"
Sep 17 17:59:17.158002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.157941    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24"} err="failed to get container status \"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24\": rpc error: code = NotFound desc = could not find container \"433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24\": container with ID starting with 433d78e989c620e5dadf6f4dc3610b217f781a7d1681cae1f948597c44b15c24 not found: ID does not exist"
Sep 17 17:59:17.158002 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.157963    2581 scope.go:117] "RemoveContainer" containerID="971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"
Sep 17 17:59:17.158234 ip-10-0-134-156 kubenswrapper[2581]: E0917 17:59:17.158216    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1\": container with ID starting with 971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1 not found: ID does not exist" containerID="971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"
Sep 17 17:59:17.158273 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.158239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1"} err="failed to get container status \"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1\": rpc error: code = NotFound desc = could not find container \"971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1\": container with ID starting with 971ad49c66c470cc3159a1caeadd102161fa2b7f2d552d37abd48f17f61ce6d1 not found: ID does not exist"
Sep 17 17:59:17.178652 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.178619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:59:17.187556 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.187538    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-69571-predictor-7f9bb65cf7-wc74s"]
Sep 17 17:59:17.361656 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.361624    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" path="/var/lib/kubelet/pods/4473f1a6-6449-4c10-97c0-68205f32cf85/volumes"
Sep 17 17:59:17.629170 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.629086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:17.721074 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721045    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 17:59:17.721347 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721335    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kube-rbac-proxy"
Sep 17 17:59:17.721390 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721348    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kube-rbac-proxy"
Sep 17 17:59:17.721390 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721365    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="storage-initializer"
Sep 17 17:59:17.721390 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="storage-initializer"
Sep 17 17:59:17.721390 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721385    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container"
Sep 17 17:59:17.721522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container"
Sep 17 17:59:17.721522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721436    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kserve-container"
Sep 17 17:59:17.721522 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.721445    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4473f1a6-6449-4c10-97c0-68205f32cf85" containerName="kube-rbac-proxy"
Sep 17 17:59:17.725873 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.725856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.730420 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.730398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-03e38-kube-rbac-proxy-sar-config\""
Sep 17 17:59:17.730558 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.730397    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:59:17.730756 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.730739    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:59:17.731339 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.731319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:17.731454 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.731337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:17.731454 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.731320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-03e38-predictor-serving-cert\""
Sep 17 17:59:17.731454 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.731431    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:59:17.765120 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.765100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 17:59:17.842242 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.842211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:17.875283 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.875258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq9n\" (UniqueName: \"kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.875576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.875306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.875576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.875385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.875576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.875412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.875576 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.875445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.976762 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.976685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq9n\" (UniqueName: \"kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.976762 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.976746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.976924 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.976793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.976924 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.976822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.976924 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.976863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.977217 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.977193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.977478 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.977460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.977515 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.977470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.979267 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.979245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:17.986612 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:17.986590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq9n\" (UniqueName: \"kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n\") pod \"raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:18.035613 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:18.035584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:18.310806 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:18.310777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 17:59:18.311978 ip-10-0-134-156 kubenswrapper[2581]: W0917 17:59:18.311950    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7481ee96_33d9_40ab_b460_17eaf4a58ffa.slice/crio-9d5513942f0d614ef01c313545053921d32cee0cc6897f58b7ad20d45bae592d WatchSource:0}: Error finding container 9d5513942f0d614ef01c313545053921d32cee0cc6897f58b7ad20d45bae592d: Status 404 returned error can't find the container with id 9d5513942f0d614ef01c313545053921d32cee0cc6897f58b7ad20d45bae592d
Sep 17 17:59:19.142723 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:19.142688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerStarted","Data":"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"}
Sep 17 17:59:19.143127 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:19.142742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerStarted","Data":"9d5513942f0d614ef01c313545053921d32cee0cc6897f58b7ad20d45bae592d"}
Sep 17 17:59:22.157060 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:22.156976    2581 generic.go:356] "Generic (PLEG): container finished" podID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerID="83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2" exitCode=0
Sep 17 17:59:22.157060 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:22.157037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerDied","Data":"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"}
Sep 17 17:59:23.164829 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.164793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerStarted","Data":"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"}
Sep 17 17:59:23.165256 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.164882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerStarted","Data":"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"}
Sep 17 17:59:23.165256 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.165153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:23.165369 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.165303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:23.166568 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.166542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:59:23.203900 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:23.203841    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podStartSLOduration=2.67429655 podStartE2EDuration="6.203827758s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.529531208s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:23.202804544 +0000 UTC m=+1102.070937510" watchObservedRunningTime="2026-09-17 17:59:23.203827758 +0000 UTC m=+1102.071960737"
Sep 17 17:59:24.169452 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:24.169405    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:59:29.173392 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:29.173363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 17:59:29.173941 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:29.173916    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:59:39.174706 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:39.174637    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:59:49.174414 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:49.174375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:59:59.174583 ip-10-0-134-156 kubenswrapper[2581]: I0917 17:59:59.174541    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:00:09.173900 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:09.173858    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:00:16.131952 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:16.131912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:19.174045 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:19.174001    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:00:21.327129 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.327098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:21.372246 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.372215    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:00:21.375567 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.375552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.380413 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.380384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:00:21.380529 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.380438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:00:21.397498 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.397477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.397599 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.397503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wqd\" (UniqueName: \"kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.397599 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.397534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.397793 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.397772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.397836 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.397821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.403066 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.403044    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:00:21.499025 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.498990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.499148 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.499048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.499148 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.499066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wqd\" (UniqueName: \"kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.499148 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.499087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.499148 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.499124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.503662 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.500012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.503662 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.500404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.503662 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.500467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.503662 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.502715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.506020 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.505993    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:00:21.511071 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.511054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.514457 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.514438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:00:21.516898 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.516879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wqd\" (UniqueName: \"kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.521160 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.521138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:00:21.600339 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.600259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.600466 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.600373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.600466 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.600403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.600466 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.600421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.600598 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.600482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9nd\" (UniqueName: \"kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.685705 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.685671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:21.692944 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.692914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:21.701263 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9nd\" (UniqueName: \"kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701348 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701402 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701445 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701483 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701680 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:00:21.701633    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:00:21.701768 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.701686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.701768 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:00:21.701733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls podName:c24cc4c4-fd14-44f6-80a7-6612a88d75ba nodeName:}" failed. No retries permitted until 2026-09-17 18:00:22.201714056 +0000 UTC m=+1161.069847012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls") pod "isvc-canary-create-predictor-5f845bd94d-vj9wm" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:00:21.702027 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.702007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.702142 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.702124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.714257 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.714235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9nd\" (UniqueName: \"kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:21.794018 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.793985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:21.896069 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.895978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:21.966015 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:21.965991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:00:21.968007 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:00:21.967983    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde34f6eb_d4a7_4efe_a527_6de22912a33c.slice/crio-80f46253a6d3942f6f5dce63b739d0fbb27262e9f44a93a4c9229e50390cdc1c WatchSource:0}: Error finding container 80f46253a6d3942f6f5dce63b739d0fbb27262e9f44a93a4c9229e50390cdc1c: Status 404 returned error can't find the container with id 80f46253a6d3942f6f5dce63b739d0fbb27262e9f44a93a4c9229e50390cdc1c
Sep 17 18:00:22.205681 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.205601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:22.207874 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.207846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-vj9wm\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:22.405066 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.405030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerStarted","Data":"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"}
Sep 17 18:00:22.405427 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.405096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerStarted","Data":"80f46253a6d3942f6f5dce63b739d0fbb27262e9f44a93a4c9229e50390cdc1c"}
Sep 17 18:00:22.428438 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.428409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:22.715573 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:22.715527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:00:22.717716 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:00:22.717687    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24cc4c4_fd14_44f6_80a7_6612a88d75ba.slice/crio-c3ad13691daf0d00c0ce318beebd52dd95893a234c588b4ca1a96fadd63a4028 WatchSource:0}: Error finding container c3ad13691daf0d00c0ce318beebd52dd95893a234c588b4ca1a96fadd63a4028: Status 404 returned error can't find the container with id c3ad13691daf0d00c0ce318beebd52dd95893a234c588b4ca1a96fadd63a4028
Sep 17 18:00:23.410069 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:23.410033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerStarted","Data":"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"}
Sep 17 18:00:23.410459 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:23.410098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerStarted","Data":"c3ad13691daf0d00c0ce318beebd52dd95893a234c588b4ca1a96fadd63a4028"}
Sep 17 18:00:26.424775 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:26.424743    2581 generic.go:356] "Generic (PLEG): container finished" podID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerID="af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2" exitCode=0
Sep 17 18:00:26.425269 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:26.424810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerDied","Data":"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"}
Sep 17 18:00:27.431201 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.431164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerStarted","Data":"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"}
Sep 17 18:00:27.431680 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.431235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerStarted","Data":"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"}
Sep 17 18:00:27.431680 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.431468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:27.432696 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.432670    2581 generic.go:356] "Generic (PLEG): container finished" podID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerID="a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e" exitCode=0
Sep 17 18:00:27.432815 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.432741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerDied","Data":"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"}
Sep 17 18:00:27.454767 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:27.454723    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podStartSLOduration=2.49974707 podStartE2EDuration="6.45471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95496293s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:27.452960658 +0000 UTC m=+1166.321093622" watchObservedRunningTime="2026-09-17 18:00:27.45471 +0000 UTC m=+1166.322842981"
Sep 17 18:00:28.438855 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:28.438820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerStarted","Data":"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"}
Sep 17 18:00:28.439273 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:28.438876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerStarted","Data":"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"}
Sep 17 18:00:28.439273 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:28.439062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:28.440402 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:28.440375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:00:28.464174 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:28.462484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podStartSLOduration=3.617662402 podStartE2EDuration="7.462466276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.844803874s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:28.461068469 +0000 UTC m=+1167.329201425" watchObservedRunningTime="2026-09-17 18:00:28.462466276 +0000 UTC m=+1167.330599240"
Sep 17 18:00:29.173961 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:29.173924    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:00:29.443327 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:29.443234    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:00:33.439476 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:33.439443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:33.439925 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:33.439863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:33.441181 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:33.441153    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:00:33.444605 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:33.444584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:00:33.459717 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:33.459685    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:00:34.447726 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:34.447699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:00:34.448159 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:34.448134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:00:34.464694 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:34.464636    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:00:39.174852 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:39.174821    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 18:00:44.039576 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:44.039536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:44.106262 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:44.106228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:44.448603 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:44.448502    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:00:44.464963 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:44.464930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:00:47.593424 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:47.593392    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 18:00:47.593839 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:47.593610    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" containerID="cri-o://f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137" gracePeriod=30
Sep 17 18:00:47.593839 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:47.593678    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kube-rbac-proxy" containerID="cri-o://df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5" gracePeriod=30
Sep 17 18:00:47.598224 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:47.597806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:48.522422 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:48.522389    2581 generic.go:356] "Generic (PLEG): container finished" podID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerID="df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5" exitCode=2
Sep 17 18:00:48.522599 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:48.522457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerDied","Data":"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"}
Sep 17 18:00:49.169886 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:49.169847    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 18:00:49.174139 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:49.174114    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:00:52.334930 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.334908    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 18:00:52.454484 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jq9n\" (UniqueName: \"kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n\") pod \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") "
Sep 17 18:00:52.454484 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls\") pod \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") "
Sep 17 18:00:52.454767 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert\") pod \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") "
Sep 17 18:00:52.454767 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location\") pod \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") "
Sep 17 18:00:52.454767 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config\") pod \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\" (UID: \"7481ee96-33d9-40ab-b460-17eaf4a58ffa\") "
Sep 17 18:00:52.454969 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454942    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7481ee96-33d9-40ab-b460-17eaf4a58ffa" (UID: "7481ee96-33d9-40ab-b460-17eaf4a58ffa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:52.455033 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.454963    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7481ee96-33d9-40ab-b460-17eaf4a58ffa" (UID: "7481ee96-33d9-40ab-b460-17eaf4a58ffa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:52.455095 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.455074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7481ee96-33d9-40ab-b460-17eaf4a58ffa" (UID: "7481ee96-33d9-40ab-b460-17eaf4a58ffa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:52.457154 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.457121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n" (OuterVolumeSpecName: "kube-api-access-6jq9n") pod "7481ee96-33d9-40ab-b460-17eaf4a58ffa" (UID: "7481ee96-33d9-40ab-b460-17eaf4a58ffa"). InnerVolumeSpecName "kube-api-access-6jq9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:52.457154 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.457126    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7481ee96-33d9-40ab-b460-17eaf4a58ffa" (UID: "7481ee96-33d9-40ab-b460-17eaf4a58ffa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:52.540876 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.540849    2581 generic.go:356] "Generic (PLEG): container finished" podID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerID="f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137" exitCode=0
Sep 17 18:00:52.541025 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.540902    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"
Sep 17 18:00:52.541025 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.540940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerDied","Data":"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"}
Sep 17 18:00:52.541025 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.541000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4" event={"ID":"7481ee96-33d9-40ab-b460-17eaf4a58ffa","Type":"ContainerDied","Data":"9d5513942f0d614ef01c313545053921d32cee0cc6897f58b7ad20d45bae592d"}
Sep 17 18:00:52.541025 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.541026    2581 scope.go:117] "RemoveContainer" containerID="df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"
Sep 17 18:00:52.549359 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.549346    2581 scope.go:117] "RemoveContainer" containerID="f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"
Sep 17 18:00:52.555808 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.555789    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jq9n\" (UniqueName: \"kubernetes.io/projected/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-api-access-6jq9n\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.555808 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.555808    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7481ee96-33d9-40ab-b460-17eaf4a58ffa-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.555927 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.555817    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.555927 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.555825    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.555927 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.555833    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7481ee96-33d9-40ab-b460-17eaf4a58ffa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.557258 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.557243    2581 scope.go:117] "RemoveContainer" containerID="83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"
Sep 17 18:00:52.563628 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.563601    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 18:00:52.566464 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.566443    2581 scope.go:117] "RemoveContainer" containerID="df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"
Sep 17 18:00:52.566952 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:00:52.566932    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5\": container with ID starting with df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5 not found: ID does not exist" containerID="df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"
Sep 17 18:00:52.567035 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.566963    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5"} err="failed to get container status \"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5\": rpc error: code = NotFound desc = could not find container \"df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5\": container with ID starting with df9398acae3634d010d7d9b01fce26c98d0c8e0598f3cff96612bf7e83dde5f5 not found: ID does not exist"
Sep 17 18:00:52.567035 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.566982    2581 scope.go:117] "RemoveContainer" containerID="f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"
Sep 17 18:00:52.567289 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:00:52.567262    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137\": container with ID starting with f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137 not found: ID does not exist" containerID="f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"
Sep 17 18:00:52.567336 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.567297    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137"} err="failed to get container status \"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137\": rpc error: code = NotFound desc = could not find container \"f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137\": container with ID starting with f7ff0d2d16ceca814cb790fed088f19e96a096e0c7c0e6c5f5e573935d02f137 not found: ID does not exist"
Sep 17 18:00:52.567336 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.567321    2581 scope.go:117] "RemoveContainer" containerID="83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"
Sep 17 18:00:52.567584 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:00:52.567565    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2\": container with ID starting with 83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2 not found: ID does not exist" containerID="83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"
Sep 17 18:00:52.567632 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.567591    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2"} err="failed to get container status \"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2\": rpc error: code = NotFound desc = could not find container \"83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2\": container with ID starting with 83211d4e89a8c9172979647421bc631f64e0002399af82fd834383c8a60922a2 not found: ID does not exist"
Sep 17 18:00:52.567763 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:52.567726    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-03e38-predictor-758f4d9dcc-kn5d4"]
Sep 17 18:00:53.362150 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:53.362114    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" path="/var/lib/kubelet/pods/7481ee96-33d9-40ab-b460-17eaf4a58ffa/volumes"
Sep 17 18:00:54.448626 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:54.448588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:00:54.465164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:00:54.465135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:03.321772 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:03.321743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 18:01:03.348394 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:03.348372    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 18:01:04.449126 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:04.449085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:01:04.465400 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:04.465359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:14.448865 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:14.448826    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:01:14.465013 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:14.464974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:24.448530 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:24.448493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:01:24.464784 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:24.464756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:34.449106 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:34.449071    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:01:34.464984 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:34.464952    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:44.449491 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:44.449418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:01:44.465822 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:44.465789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:01:51.279910 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.279873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:51.383983 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.383950    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:01:51.384271 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.384229    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" containerID="cri-o://b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58" gracePeriod=30
Sep 17 18:01:51.384464 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.384436    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kube-rbac-proxy" containerID="cri-o://41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337" gracePeriod=30
Sep 17 18:01:51.387892 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.387869    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:01:51.388095 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.388060    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" containerID="cri-o://24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf" gracePeriod=30
Sep 17 18:01:51.388172 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.388138    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kube-rbac-proxy" containerID="cri-o://a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715" gracePeriod=30
Sep 17 18:01:51.396562 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.396536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:51.462096 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.462061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:51.790813 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.790785    2581 generic.go:356] "Generic (PLEG): container finished" podID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerID="a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715" exitCode=2
Sep 17 18:01:51.790986 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.790852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerDied","Data":"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"}
Sep 17 18:01:51.793263 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.793241    2581 generic.go:356] "Generic (PLEG): container finished" podID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerID="41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337" exitCode=2
Sep 17 18:01:51.793385 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:51.793277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerDied","Data":"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"}
Sep 17 18:01:53.445825 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:53.445786    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 18:01:54.444062 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:54.444015    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 18:01:54.448254 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:54.448232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 18:01:54.465494 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:54.465461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:01:56.133369 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.133346    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:01:56.235700 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.235678    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:01:56.271238 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271212    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location\") pod \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") "
Sep 17 18:01:56.271410 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271320    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config\") pod \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") "
Sep 17 18:01:56.271410 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271354    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert\") pod \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") "
Sep 17 18:01:56.271410 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w9nd\" (UniqueName: \"kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd\") pod \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") "
Sep 17 18:01:56.271566 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") pod \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\" (UID: \"c24cc4c4-fd14-44f6-80a7-6612a88d75ba\") "
Sep 17 18:01:56.271566 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c24cc4c4-fd14-44f6-80a7-6612a88d75ba" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:56.271772 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271754    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.271836 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c24cc4c4-fd14-44f6-80a7-6612a88d75ba" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:56.271836 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.271815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c24cc4c4-fd14-44f6-80a7-6612a88d75ba" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:56.273574 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.273551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c24cc4c4-fd14-44f6-80a7-6612a88d75ba" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:56.273681 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.273550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd" (OuterVolumeSpecName: "kube-api-access-8w9nd") pod "c24cc4c4-fd14-44f6-80a7-6612a88d75ba" (UID: "c24cc4c4-fd14-44f6-80a7-6612a88d75ba"). InnerVolumeSpecName "kube-api-access-8w9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:56.372427 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location\") pod \"de34f6eb-d4a7-4efe-a527-6de22912a33c\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") "
Sep 17 18:01:56.372427 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config\") pod \"de34f6eb-d4a7-4efe-a527-6de22912a33c\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") "
Sep 17 18:01:56.372716 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert\") pod \"de34f6eb-d4a7-4efe-a527-6de22912a33c\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") "
Sep 17 18:01:56.372716 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls\") pod \"de34f6eb-d4a7-4efe-a527-6de22912a33c\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") "
Sep 17 18:01:56.372716 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6wqd\" (UniqueName: \"kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd\") pod \"de34f6eb-d4a7-4efe-a527-6de22912a33c\" (UID: \"de34f6eb-d4a7-4efe-a527-6de22912a33c\") "
Sep 17 18:01:56.372874 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de34f6eb-d4a7-4efe-a527-6de22912a33c" (UID: "de34f6eb-d4a7-4efe-a527-6de22912a33c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:56.372874 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de34f6eb-d4a7-4efe-a527-6de22912a33c" (UID: "de34f6eb-d4a7-4efe-a527-6de22912a33c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:56.372874 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372860    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.372993 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372878    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.372993 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372893    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9nd\" (UniqueName: \"kubernetes.io/projected/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-kube-api-access-8w9nd\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.372993 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372908    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c24cc4c4-fd14-44f6-80a7-6612a88d75ba-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.372993 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.372914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de34f6eb-d4a7-4efe-a527-6de22912a33c" (UID: "de34f6eb-d4a7-4efe-a527-6de22912a33c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:56.374684 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.374633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de34f6eb-d4a7-4efe-a527-6de22912a33c" (UID: "de34f6eb-d4a7-4efe-a527-6de22912a33c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:56.374773 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.374696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd" (OuterVolumeSpecName: "kube-api-access-l6wqd") pod "de34f6eb-d4a7-4efe-a527-6de22912a33c" (UID: "de34f6eb-d4a7-4efe-a527-6de22912a33c"). InnerVolumeSpecName "kube-api-access-l6wqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:56.457066 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.457019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:56.474041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.473971    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wqd\" (UniqueName: \"kubernetes.io/projected/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-api-access-l6wqd\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.474041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.473997    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de34f6eb-d4a7-4efe-a527-6de22912a33c-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.474041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.474011    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.474041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.474024    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de34f6eb-d4a7-4efe-a527-6de22912a33c-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.474041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.474036    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de34f6eb-d4a7-4efe-a527-6de22912a33c-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.525884 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.525853    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:01:56.526260 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526243    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="storage-initializer"
Sep 17 18:01:56.526260 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526262    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="storage-initializer"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526273    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526281    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526291    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="storage-initializer"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526299    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="storage-initializer"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526323    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kube-rbac-proxy"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526332    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kube-rbac-proxy"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526349    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="storage-initializer"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526358    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="storage-initializer"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526369    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526378    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526389    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526397    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526406    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kube-rbac-proxy"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526414    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kube-rbac-proxy"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kube-rbac-proxy"
Sep 17 18:01:56.526440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kube-rbac-proxy"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526522    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kserve-container"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526533    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kube-rbac-proxy"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526543    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kserve-container"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526552    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerName="kube-rbac-proxy"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526561    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerName="kube-rbac-proxy"
Sep 17 18:01:56.527256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.526569    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7481ee96-33d9-40ab-b460-17eaf4a58ffa" containerName="kserve-container"
Sep 17 18:01:56.531048 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.531030    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:01:56.531165 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.531147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.534089 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.534068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:01:56.534174 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.534068    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:01:56.534412 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.534396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.537075 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.537059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:01:56.543120 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.543101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:01:56.544774 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.544754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:01:56.546531 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.546505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:56.635297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.635255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:56.675118 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.675297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.675297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.675297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.675297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.675297 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.675470 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcnb\" (UniqueName: \"kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.675470 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv52\" (UniqueName: \"kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.675470 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.675470 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.675420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcnb\" (UniqueName: \"kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.776406 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv52\" (UniqueName: \"kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.776798 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.776745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.777078 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.777049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.777158 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.777058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.777158 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.777057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.777158 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.777112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.777158 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.777154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.778739 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.778721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.778969 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.778954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.788169 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.788152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv52\" (UniqueName: \"kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52\") pod \"isvc-canary-promote-predictor-567686cc76-rg47w\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.788243 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.788225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcnb\" (UniqueName: \"kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.815507 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.815478    2581 generic.go:356] "Generic (PLEG): container finished" podID="de34f6eb-d4a7-4efe-a527-6de22912a33c" containerID="b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58" exitCode=0
Sep 17 18:01:56.815633 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.815536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"
Sep 17 18:01:56.815633 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.815564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerDied","Data":"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"}
Sep 17 18:01:56.815633 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.815630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg" event={"ID":"de34f6eb-d4a7-4efe-a527-6de22912a33c","Type":"ContainerDied","Data":"80f46253a6d3942f6f5dce63b739d0fbb27262e9f44a93a4c9229e50390cdc1c"}
Sep 17 18:01:56.815809 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.815691    2581 scope.go:117] "RemoveContainer" containerID="41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"
Sep 17 18:01:56.817675 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.817626    2581 generic.go:356] "Generic (PLEG): container finished" podID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" containerID="24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf" exitCode=0
Sep 17 18:01:56.817789 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.817697    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"
Sep 17 18:01:56.817789 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.817722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerDied","Data":"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"}
Sep 17 18:01:56.817789 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.817774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm" event={"ID":"c24cc4c4-fd14-44f6-80a7-6612a88d75ba","Type":"ContainerDied","Data":"c3ad13691daf0d00c0ce318beebd52dd95893a234c588b4ca1a96fadd63a4028"}
Sep 17 18:01:56.824293 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.824175    2581 scope.go:117] "RemoveContainer" containerID="b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"
Sep 17 18:01:56.831439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.831419    2581 scope.go:117] "RemoveContainer" containerID="af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"
Sep 17 18:01:56.838585 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.838570    2581 scope.go:117] "RemoveContainer" containerID="41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"
Sep 17 18:01:56.838871 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.838856    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337\": container with ID starting with 41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337 not found: ID does not exist" containerID="41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"
Sep 17 18:01:56.838921 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.838880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337"} err="failed to get container status \"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337\": rpc error: code = NotFound desc = could not find container \"41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337\": container with ID starting with 41380a43381aa4b0543a6ddb23be7efc899decd4655058429cc8720ef8a32337 not found: ID does not exist"
Sep 17 18:01:56.838921 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.838895    2581 scope.go:117] "RemoveContainer" containerID="b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"
Sep 17 18:01:56.839152 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.839134    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58\": container with ID starting with b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58 not found: ID does not exist" containerID="b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"
Sep 17 18:01:56.839200 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.839157    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58"} err="failed to get container status \"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58\": rpc error: code = NotFound desc = could not find container \"b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58\": container with ID starting with b6ca032f88f2872185f84c3522bf665d2d30ef395f763a7362c2bf9c746bcc58 not found: ID does not exist"
Sep 17 18:01:56.839200 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.839173    2581 scope.go:117] "RemoveContainer" containerID="af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"
Sep 17 18:01:56.839385 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.839370    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2\": container with ID starting with af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2 not found: ID does not exist" containerID="af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"
Sep 17 18:01:56.839425 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.839397    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2"} err="failed to get container status \"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2\": rpc error: code = NotFound desc = could not find container \"af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2\": container with ID starting with af4aa8bf14dbdd4e5243882911acb2730fa6c16e5b9c06ed2bb4fa210508d7a2 not found: ID does not exist"
Sep 17 18:01:56.839425 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.839413    2581 scope.go:117] "RemoveContainer" containerID="a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"
Sep 17 18:01:56.842514 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.842493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:01:56.846537 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.846520    2581 scope.go:117] "RemoveContainer" containerID="24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"
Sep 17 18:01:56.847242 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.847222    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:01:56.850423 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.850333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:01:56.852260 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.852240    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-ps8sg"]
Sep 17 18:01:56.855090 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.855073    2581 scope.go:117] "RemoveContainer" containerID="a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"
Sep 17 18:01:56.863167 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.863134    2581 scope.go:117] "RemoveContainer" containerID="a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"
Sep 17 18:01:56.863458 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.863433    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715\": container with ID starting with a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715 not found: ID does not exist" containerID="a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"
Sep 17 18:01:56.863558 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.863463    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715"} err="failed to get container status \"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715\": rpc error: code = NotFound desc = could not find container \"a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715\": container with ID starting with a2aff1acaab1bbe83c545d1eb623b7722e427d47de9206a015e2c2a23b9cf715 not found: ID does not exist"
Sep 17 18:01:56.863558 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.863479    2581 scope.go:117] "RemoveContainer" containerID="24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"
Sep 17 18:01:56.863864 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.863841    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf\": container with ID starting with 24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf not found: ID does not exist" containerID="24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"
Sep 17 18:01:56.863952 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.863872    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf"} err="failed to get container status \"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf\": rpc error: code = NotFound desc = could not find container \"24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf\": container with ID starting with 24f443ace2b0661e6622b789627c1802093a62784ed3ffa62d72d98766e7a8cf not found: ID does not exist"
Sep 17 18:01:56.863952 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.863895    2581 scope.go:117] "RemoveContainer" containerID="a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"
Sep 17 18:01:56.864406 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:01:56.864322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e\": container with ID starting with a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e not found: ID does not exist" containerID="a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"
Sep 17 18:01:56.864633 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.864406    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e"} err="failed to get container status \"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e\": rpc error: code = NotFound desc = could not find container \"a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e\": container with ID starting with a8eeb1901f2e8eca1750fc0fda907cc7057cedc99af29900e31e6a9e59805e7e not found: ID does not exist"
Sep 17 18:01:56.866686 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.866664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:01:56.871289 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:56.871269    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-vj9wm"]
Sep 17 18:01:57.149686 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.149633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:01:57.154043 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:01:57.153994    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04daccb3_c852_490c_94da_1a828bca67f2.slice/crio-ca0654d8fdc1e3918c5309cd648da9d09b216bc252e9b50af7c32bbc7b18cedd WatchSource:0}: Error finding container ca0654d8fdc1e3918c5309cd648da9d09b216bc252e9b50af7c32bbc7b18cedd: Status 404 returned error can't find the container with id ca0654d8fdc1e3918c5309cd648da9d09b216bc252e9b50af7c32bbc7b18cedd
Sep 17 18:01:57.172843 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.172817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:01:57.172957 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:01:57.172937    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2fc8ed_c452_4ed3_ab6a_f7e896545779.slice/crio-0a5c87232834b052fcf5960b1882457f3698791c6bdc7d0a2d7afc86a2ba2334 WatchSource:0}: Error finding container 0a5c87232834b052fcf5960b1882457f3698791c6bdc7d0a2d7afc86a2ba2334: Status 404 returned error can't find the container with id 0a5c87232834b052fcf5960b1882457f3698791c6bdc7d0a2d7afc86a2ba2334
Sep 17 18:01:57.362065 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.362026    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24cc4c4-fd14-44f6-80a7-6612a88d75ba" path="/var/lib/kubelet/pods/c24cc4c4-fd14-44f6-80a7-6612a88d75ba/volumes"
Sep 17 18:01:57.362839 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.362814    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de34f6eb-d4a7-4efe-a527-6de22912a33c" path="/var/lib/kubelet/pods/de34f6eb-d4a7-4efe-a527-6de22912a33c/volumes"
Sep 17 18:01:57.825657 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.825606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerStarted","Data":"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"}
Sep 17 18:01:57.825856 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.825701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerStarted","Data":"ca0654d8fdc1e3918c5309cd648da9d09b216bc252e9b50af7c32bbc7b18cedd"}
Sep 17 18:01:57.827429 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.827394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerStarted","Data":"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"}
Sep 17 18:01:57.827591 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:57.827449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerStarted","Data":"0a5c87232834b052fcf5960b1882457f3698791c6bdc7d0a2d7afc86a2ba2334"}
Sep 17 18:01:59.276355 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:01:59.276322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:01.846750 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:01.846719    2581 generic.go:356] "Generic (PLEG): container finished" podID="04daccb3-c852-490c-94da-1a828bca67f2" containerID="3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca" exitCode=0
Sep 17 18:02:01.847200 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:01.846798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerDied","Data":"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"}
Sep 17 18:02:01.848359 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:01.848340    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerID="d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482" exitCode=0
Sep 17 18:02:01.848428 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:01.848389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerDied","Data":"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"}
Sep 17 18:02:02.854821 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.854775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerStarted","Data":"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"}
Sep 17 18:02:02.855321 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.854884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerStarted","Data":"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"}
Sep 17 18:02:02.855321 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.855203    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:02:02.855466 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.855370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:02:02.856990 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.856963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:02.857556 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.857535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerStarted","Data":"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"}
Sep 17 18:02:02.857697 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.857588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerStarted","Data":"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"}
Sep 17 18:02:02.857808 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.857791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:02:02.890101 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.890054    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podStartSLOduration=2.8614187700000002 podStartE2EDuration="6.890037347s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.028618577s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:02.888573624 +0000 UTC m=+1261.756706583" watchObservedRunningTime="2026-09-17 18:02:02.890037347 +0000 UTC m=+1261.758170310"
Sep 17 18:02:02.917809 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:02.917757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podStartSLOduration=3.301260977 podStartE2EDuration="6.917741655s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.616480678s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:02.914867345 +0000 UTC m=+1261.783000309" watchObservedRunningTime="2026-09-17 18:02:02.917741655 +0000 UTC m=+1261.785874617"
Sep 17 18:02:03.861706 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:03.861669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:02:03.862193 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:03.861757    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:03.863384 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:03.863354    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:04.866441 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:04.866403    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:08.866032 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:08.866003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:02:08.866531 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:08.866504    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:09.870661 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:09.870620    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:02:09.871293 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:09.871266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:18.866603 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:18.866564    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:19.871597 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:19.871556    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:28.867070 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:28.867020    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:29.871606 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:29.871565    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:38.866613 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:38.866563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:39.871739 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:39.871692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:48.867364 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:48.867323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:49.871389 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:49.871350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:02:58.867446 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:58.867403    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:02:59.871779 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:02:59.871740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:03:08.866531 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:08.866489    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:03:09.871954 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:09.871909    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:03:18.867878 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:18.867790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:03:19.034629 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.034597    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:03:19.034875 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.034832    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container" containerID="cri-o://043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f" gracePeriod=30
Sep 17 18:03:19.034922 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.034892    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kube-rbac-proxy" containerID="cri-o://b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c" gracePeriod=30
Sep 17 18:03:19.036192 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.036172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:03:19.171785 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.171755    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerID="b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c" exitCode=2
Sep 17 18:03:19.171923 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.171822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerDied","Data":"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"}
Sep 17 18:03:19.867226 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:19.867180    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:03:23.571650 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.571611    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:03:23.616931 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.616870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgv52\" (UniqueName: \"kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52\") pod \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") "
Sep 17 18:03:23.616931 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.616908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert\") pod \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") "
Sep 17 18:03:23.617121 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.616943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config\") pod \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") "
Sep 17 18:03:23.617121 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.616961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls\") pod \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") "
Sep 17 18:03:23.617121 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.617014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location\") pod \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\" (UID: \"7a2fc8ed-c452-4ed3-ab6a-f7e896545779\") "
Sep 17 18:03:23.617485 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.617324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a2fc8ed-c452-4ed3-ab6a-f7e896545779" (UID: "7a2fc8ed-c452-4ed3-ab6a-f7e896545779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:23.617485 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.617332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a2fc8ed-c452-4ed3-ab6a-f7e896545779" (UID: "7a2fc8ed-c452-4ed3-ab6a-f7e896545779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:23.617485 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.617467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a2fc8ed-c452-4ed3-ab6a-f7e896545779" (UID: "7a2fc8ed-c452-4ed3-ab6a-f7e896545779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:23.619132 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.619106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52" (OuterVolumeSpecName: "kube-api-access-tgv52") pod "7a2fc8ed-c452-4ed3-ab6a-f7e896545779" (UID: "7a2fc8ed-c452-4ed3-ab6a-f7e896545779"). InnerVolumeSpecName "kube-api-access-tgv52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:23.619256 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.619214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a2fc8ed-c452-4ed3-ab6a-f7e896545779" (UID: "7a2fc8ed-c452-4ed3-ab6a-f7e896545779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:23.718164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.718127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgv52\" (UniqueName: \"kubernetes.io/projected/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-api-access-tgv52\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:03:23.718164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.718156    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:03:23.718164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.718165    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:03:23.718374 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.718174    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:03:23.718374 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:23.718183    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a2fc8ed-c452-4ed3-ab6a-f7e896545779-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:03:24.201914 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.201879    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerID="043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f" exitCode=0
Sep 17 18:03:24.202088 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.201930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerDied","Data":"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"}
Sep 17 18:03:24.202088 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.201933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"
Sep 17 18:03:24.202088 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.201978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w" event={"ID":"7a2fc8ed-c452-4ed3-ab6a-f7e896545779","Type":"ContainerDied","Data":"0a5c87232834b052fcf5960b1882457f3698791c6bdc7d0a2d7afc86a2ba2334"}
Sep 17 18:03:24.202088 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.202002    2581 scope.go:117] "RemoveContainer" containerID="b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"
Sep 17 18:03:24.212440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.212415    2581 scope.go:117] "RemoveContainer" containerID="043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"
Sep 17 18:03:24.219698 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.219679    2581 scope.go:117] "RemoveContainer" containerID="d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"
Sep 17 18:03:24.226388 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.226369    2581 scope.go:117] "RemoveContainer" containerID="b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"
Sep 17 18:03:24.226673 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:03:24.226633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c\": container with ID starting with b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c not found: ID does not exist" containerID="b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"
Sep 17 18:03:24.226799 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.226681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c"} err="failed to get container status \"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c\": rpc error: code = NotFound desc = could not find container \"b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c\": container with ID starting with b40dcb6da7e28e36f56f0b2e96f4413cdb5d781fbcbb5a340834acba05c6e69c not found: ID does not exist"
Sep 17 18:03:24.226799 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.226704    2581 scope.go:117] "RemoveContainer" containerID="043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"
Sep 17 18:03:24.227079 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:03:24.227033    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f\": container with ID starting with 043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f not found: ID does not exist" containerID="043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"
Sep 17 18:03:24.227079 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.227061    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f"} err="failed to get container status \"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f\": rpc error: code = NotFound desc = could not find container \"043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f\": container with ID starting with 043bb0741dab31e31da25b878153c2f451d9276da8f912d69a0a1d7446e93b3f not found: ID does not exist"
Sep 17 18:03:24.227238 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.227082    2581 scope.go:117] "RemoveContainer" containerID="d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"
Sep 17 18:03:24.227448 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:03:24.227380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482\": container with ID starting with d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482 not found: ID does not exist" containerID="d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"
Sep 17 18:03:24.227448 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.227408    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482"} err="failed to get container status \"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482\": rpc error: code = NotFound desc = could not find container \"d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482\": container with ID starting with d64371bbe431745ee7d6110ad7ca1c146611f9116883f421d8c3a4d953b44482 not found: ID does not exist"
Sep 17 18:03:24.229501 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.229480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:03:24.233289 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:24.233268    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-rg47w"]
Sep 17 18:03:25.360531 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:25.360495    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" path="/var/lib/kubelet/pods/7a2fc8ed-c452-4ed3-ab6a-f7e896545779/volumes"
Sep 17 18:03:26.667737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:03:26.667700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.869967 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:56.869934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:04:56.870453 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:56.870219    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" containerID="cri-o://1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e" gracePeriod=30
Sep 17 18:04:56.870453 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:56.870246    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kube-rbac-proxy" containerID="cri-o://5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4" gracePeriod=30
Sep 17 18:04:56.908727 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:56.908700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:57.571357 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:57.571325    2581 generic.go:356] "Generic (PLEG): container finished" podID="04daccb3-c852-490c-94da-1a828bca67f2" containerID="5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4" exitCode=2
Sep 17 18:04:57.571522 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:57.571389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerDied","Data":"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"}
Sep 17 18:04:58.862262 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:58.862219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 18:04:58.866570 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:04:58.866547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:05:01.513415 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.513385    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:05:01.547558 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location\") pod \"04daccb3-c852-490c-94da-1a828bca67f2\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") "
Sep 17 18:05:01.547558 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls\") pod \"04daccb3-c852-490c-94da-1a828bca67f2\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") "
Sep 17 18:05:01.547782 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547594    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpcnb\" (UniqueName: \"kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb\") pod \"04daccb3-c852-490c-94da-1a828bca67f2\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") "
Sep 17 18:05:01.547782 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config\") pod \"04daccb3-c852-490c-94da-1a828bca67f2\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") "
Sep 17 18:05:01.547782 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert\") pod \"04daccb3-c852-490c-94da-1a828bca67f2\" (UID: \"04daccb3-c852-490c-94da-1a828bca67f2\") "
Sep 17 18:05:01.547988 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.547887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04daccb3-c852-490c-94da-1a828bca67f2" (UID: "04daccb3-c852-490c-94da-1a828bca67f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:01.548077 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.548030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04daccb3-c852-490c-94da-1a828bca67f2" (UID: "04daccb3-c852-490c-94da-1a828bca67f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:01.548166 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.548140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04daccb3-c852-490c-94da-1a828bca67f2" (UID: "04daccb3-c852-490c-94da-1a828bca67f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:01.549837 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.549808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04daccb3-c852-490c-94da-1a828bca67f2" (UID: "04daccb3-c852-490c-94da-1a828bca67f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:01.549936 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.549869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb" (OuterVolumeSpecName: "kube-api-access-rpcnb") pod "04daccb3-c852-490c-94da-1a828bca67f2" (UID: "04daccb3-c852-490c-94da-1a828bca67f2"). InnerVolumeSpecName "kube-api-access-rpcnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:01.588368 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.588334    2581 generic.go:356] "Generic (PLEG): container finished" podID="04daccb3-c852-490c-94da-1a828bca67f2" containerID="1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e" exitCode=0
Sep 17 18:05:01.588540 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.588388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"
Sep 17 18:05:01.588540 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.588398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerDied","Data":"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"}
Sep 17 18:05:01.588540 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.588447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9" event={"ID":"04daccb3-c852-490c-94da-1a828bca67f2","Type":"ContainerDied","Data":"ca0654d8fdc1e3918c5309cd648da9d09b216bc252e9b50af7c32bbc7b18cedd"}
Sep 17 18:05:01.588540 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.588473    2581 scope.go:117] "RemoveContainer" containerID="5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"
Sep 17 18:05:01.597033 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.597016    2581 scope.go:117] "RemoveContainer" containerID="1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"
Sep 17 18:05:01.604316 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.604300    2581 scope.go:117] "RemoveContainer" containerID="3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"
Sep 17 18:05:01.611814 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.611794    2581 scope.go:117] "RemoveContainer" containerID="5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"
Sep 17 18:05:01.612114 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:05:01.612095    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4\": container with ID starting with 5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4 not found: ID does not exist" containerID="5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"
Sep 17 18:05:01.612194 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612134    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4"} err="failed to get container status \"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4\": rpc error: code = NotFound desc = could not find container \"5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4\": container with ID starting with 5159cc0a86ad162fd5db1e458f99273c23042fde876a13f939150fc2dd308da4 not found: ID does not exist"
Sep 17 18:05:01.612194 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612159    2581 scope.go:117] "RemoveContainer" containerID="1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"
Sep 17 18:05:01.612287 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:05:01.612429 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:05:01.612412    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e\": container with ID starting with 1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e not found: ID does not exist" containerID="1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"
Sep 17 18:05:01.612473 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612436    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e"} err="failed to get container status \"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e\": rpc error: code = NotFound desc = could not find container \"1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e\": container with ID starting with 1ecec841163d64a2804c51d79f1548f17ea0f87ccd1616f7f3193aaeef95fe2e not found: ID does not exist"
Sep 17 18:05:01.612473 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612453    2581 scope.go:117] "RemoveContainer" containerID="3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"
Sep 17 18:05:01.612823 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:05:01.612806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca\": container with ID starting with 3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca not found: ID does not exist" containerID="3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"
Sep 17 18:05:01.612869 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.612827    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca"} err="failed to get container status \"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca\": rpc error: code = NotFound desc = could not find container \"3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca\": container with ID starting with 3badb0b72aa57d671921d4b5ccb28ca5dd49fb8cfd42c45e947b5bdd676a8eca not found: ID does not exist"
Sep 17 18:05:01.617957 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.617934    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-ct9x9"]
Sep 17 18:05:01.648723 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.648701    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcnb\" (UniqueName: \"kubernetes.io/projected/04daccb3-c852-490c-94da-1a828bca67f2-kube-api-access-rpcnb\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.648835 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.648726    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.648835 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.648741    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04daccb3-c852-490c-94da-1a828bca67f2-cabundle-cert\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.648835 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.648753    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04daccb3-c852-490c-94da-1a828bca67f2-kserve-provision-location\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.648835 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:01.648764    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04daccb3-c852-490c-94da-1a828bca67f2-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:05:03.361370 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:03.361336    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04daccb3-c852-490c-94da-1a828bca67f2" path="/var/lib/kubelet/pods/04daccb3-c852-490c-94da-1a828bca67f2/volumes"
Sep 17 18:05:11.134041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.134004    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:11.182236 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182208    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:05:11.182517 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kube-rbac-proxy"
Sep 17 18:05:11.182559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182519    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kube-rbac-proxy"
Sep 17 18:05:11.182559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182537    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="storage-initializer"
Sep 17 18:05:11.182559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182542    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="storage-initializer"
Sep 17 18:05:11.182559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182549    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="storage-initializer"
Sep 17 18:05:11.182559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182554    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="storage-initializer"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182561    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182568    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182580    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kube-rbac-proxy"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kube-rbac-proxy"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182596    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182654    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kube-rbac-proxy"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04daccb3-c852-490c-94da-1a828bca67f2" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182677    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kserve-container"
Sep 17 18:05:11.182737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.182683    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2fc8ed-c452-4ed3-ab6a-f7e896545779" containerName="kube-rbac-proxy"
Sep 17 18:05:11.187318 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.187301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.191061 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.191040    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:05:11.192683 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.192662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 18:05:11.192769 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.192662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:05:11.192769 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.192663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 18:05:11.192769 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.192666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 18:05:11.204309 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.204287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:05:11.224448 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.224424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftzk\" (UniqueName: \"kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.224568 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.224482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.224568 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.224518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.235895 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.235873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:11.325653 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.325614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftzk\" (UniqueName: \"kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.325816 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.325678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.325816 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.325697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.326316 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.326286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.328074 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.328054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.338283 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.338265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftzk\" (UniqueName: \"kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-d525w\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.497623 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.497534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:11.776067 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:11.775986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:05:11.778927 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:05:11.778902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeda331a_368f_4d00_8d4b_0fee70b77bb7.slice/crio-362e8f2c9e78b0cae9c91b074c8ee1dc192d00625afd887e8ebae7e0fe423841 WatchSource:0}: Error finding container 362e8f2c9e78b0cae9c91b074c8ee1dc192d00625afd887e8ebae7e0fe423841: Status 404 returned error can't find the container with id 362e8f2c9e78b0cae9c91b074c8ee1dc192d00625afd887e8ebae7e0fe423841
Sep 17 18:05:12.637073 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.637033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerStarted","Data":"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"}
Sep 17 18:05:12.637439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.637119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerStarted","Data":"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"}
Sep 17 18:05:12.637439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.637150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerStarted","Data":"362e8f2c9e78b0cae9c91b074c8ee1dc192d00625afd887e8ebae7e0fe423841"}
Sep 17 18:05:12.637439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.637174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:12.637439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.637197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:12.639168 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.639148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:12.659462 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:12.659413    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" podStartSLOduration=1.6594004340000001 podStartE2EDuration="1.659400434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:12.658444582 +0000 UTC m=+1451.526577548" watchObservedRunningTime="2026-09-17 18:05:12.659400434 +0000 UTC m=+1451.527533398"
Sep 17 18:05:19.649388 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:19.649356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:05:21.081754 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:21.081722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:21.254432 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:21.254399    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:44.009490 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:05:44.009456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:03.346813 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:03.346783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 18:06:03.373438 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:03.373413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 18:06:35.023426 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:35.023389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:56.094759 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.094729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-d525w_aeda331a-368f-4d00-8d4b-0fee70b77bb7/kserve-container/0.log"
Sep 17 18:06:56.470966 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.470880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:56.481067 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.481038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:06:56.481264 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.481228    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kserve-container" containerID="cri-o://14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b" gracePeriod=30
Sep 17 18:06:56.481394 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.481280    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kube-rbac-proxy" containerID="cri-o://08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac" gracePeriod=30
Sep 17 18:06:56.564140 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.564113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:56.715699 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.715675    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:06:56.799150 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.799121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config\") pod \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") "
Sep 17 18:06:56.799370 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.799163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kftzk\" (UniqueName: \"kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk\") pod \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") "
Sep 17 18:06:56.799370 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.799199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls\") pod \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\" (UID: \"aeda331a-368f-4d00-8d4b-0fee70b77bb7\") "
Sep 17 18:06:56.799559 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.799531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aeda331a-368f-4d00-8d4b-0fee70b77bb7" (UID: "aeda331a-368f-4d00-8d4b-0fee70b77bb7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:56.801330 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.801300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk" (OuterVolumeSpecName: "kube-api-access-kftzk") pod "aeda331a-368f-4d00-8d4b-0fee70b77bb7" (UID: "aeda331a-368f-4d00-8d4b-0fee70b77bb7"). InnerVolumeSpecName "kube-api-access-kftzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:56.801440 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.801331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aeda331a-368f-4d00-8d4b-0fee70b77bb7" (UID: "aeda331a-368f-4d00-8d4b-0fee70b77bb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:56.899896 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.899865    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:06:56.899896 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.899891    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftzk\" (UniqueName: \"kubernetes.io/projected/aeda331a-368f-4d00-8d4b-0fee70b77bb7-kube-api-access-kftzk\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:06:56.899896 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:56.899901    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeda331a-368f-4d00-8d4b-0fee70b77bb7-proxy-tls\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:06:57.057937 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.057858    2581 generic.go:356] "Generic (PLEG): container finished" podID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerID="08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac" exitCode=2
Sep 17 18:06:57.057937 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.057882    2581 generic.go:356] "Generic (PLEG): container finished" podID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerID="14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b" exitCode=2
Sep 17 18:06:57.057937 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.057907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"
Sep 17 18:06:57.058146 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.057938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerDied","Data":"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"}
Sep 17 18:06:57.058146 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.058009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerDied","Data":"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"}
Sep 17 18:06:57.058146 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.058042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w" event={"ID":"aeda331a-368f-4d00-8d4b-0fee70b77bb7","Type":"ContainerDied","Data":"362e8f2c9e78b0cae9c91b074c8ee1dc192d00625afd887e8ebae7e0fe423841"}
Sep 17 18:06:57.058146 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.058077    2581 scope.go:117] "RemoveContainer" containerID="08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"
Sep 17 18:06:57.066538 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.066522    2581 scope.go:117] "RemoveContainer" containerID="14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"
Sep 17 18:06:57.073556 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.073538    2581 scope.go:117] "RemoveContainer" containerID="08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"
Sep 17 18:06:57.073818 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:06:57.073800    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac\": container with ID starting with 08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac not found: ID does not exist" containerID="08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"
Sep 17 18:06:57.073864 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.073828    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"} err="failed to get container status \"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac\": rpc error: code = NotFound desc = could not find container \"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac\": container with ID starting with 08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac not found: ID does not exist"
Sep 17 18:06:57.073864 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.073845    2581 scope.go:117] "RemoveContainer" containerID="14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"
Sep 17 18:06:57.074093 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:06:57.074069    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b\": container with ID starting with 14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b not found: ID does not exist" containerID="14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"
Sep 17 18:06:57.074093 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.074096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"} err="failed to get container status \"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b\": rpc error: code = NotFound desc = could not find container \"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b\": container with ID starting with 14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b not found: ID does not exist"
Sep 17 18:06:57.074282 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.074110    2581 scope.go:117] "RemoveContainer" containerID="08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"
Sep 17 18:06:57.074383 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.074364    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac"} err="failed to get container status \"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac\": rpc error: code = NotFound desc = could not find container \"08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac\": container with ID starting with 08ba2c9f1a18eb82cc9d8206c7bbe0d130229b6ad0228ed977497e9031d296ac not found: ID does not exist"
Sep 17 18:06:57.074430 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.074384    2581 scope.go:117] "RemoveContainer" containerID="14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"
Sep 17 18:06:57.074611 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.074595    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b"} err="failed to get container status \"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b\": rpc error: code = NotFound desc = could not find container \"14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b\": container with ID starting with 14c5bcdcb310e839c13209f0a2dd82db891b3480a92568ef81e2f7a0bfbd448b not found: ID does not exist"
Sep 17 18:06:57.092145 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.092118    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:06:57.096979 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.096937    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-d525w"]
Sep 17 18:06:57.100696 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.100676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:57.101104 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101089    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k6t2h/must-gather-xr525"]
Sep 17 18:06:57.101374 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101363    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kserve-container"
Sep 17 18:06:57.101414 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101375    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kserve-container"
Sep 17 18:06:57.101414 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kube-rbac-proxy"
Sep 17 18:06:57.101414 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101407    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kube-rbac-proxy"
Sep 17 18:06:57.101517 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101455    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kserve-container"
Sep 17 18:06:57.101517 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.101463    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" containerName="kube-rbac-proxy"
Sep 17 18:06:57.105504 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.105488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.108828 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.108811    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k6t2h\"/\"default-dockercfg-fcfhw\""
Sep 17 18:06:57.109311 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.109297    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6t2h\"/\"kube-root-ca.crt\""
Sep 17 18:06:57.109598 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.109584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6t2h\"/\"openshift-service-ca.crt\""
Sep 17 18:06:57.114600 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.114581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6t2h/must-gather-xr525"]
Sep 17 18:06:57.182156 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.182122    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:57.202066 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.202045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.202166 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.202098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h227\" (UniqueName: \"kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.303100 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.303055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h227\" (UniqueName: \"kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.303255 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.303118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.303441 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.303426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.312701 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.312629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h227\" (UniqueName: \"kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227\") pod \"must-gather-xr525\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") " pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.361146 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.361112    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeda331a-368f-4d00-8d4b-0fee70b77bb7" path="/var/lib/kubelet/pods/aeda331a-368f-4d00-8d4b-0fee70b77bb7/volumes"
Sep 17 18:06:57.414696 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.414669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:06:57.690295 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.690272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6t2h/must-gather-xr525"]
Sep 17 18:06:57.692255 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:06:57.692227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b92e95_3075_4f31_8824_ffe812e56c66.slice/crio-ed448b86e3e3d3000db4a8402a361d256eef1df511fea253fad07a312241147c WatchSource:0}: Error finding container ed448b86e3e3d3000db4a8402a361d256eef1df511fea253fad07a312241147c: Status 404 returned error can't find the container with id ed448b86e3e3d3000db4a8402a361d256eef1df511fea253fad07a312241147c
Sep 17 18:06:57.694008 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:57.693988    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:06:58.063420 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:06:58.063380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6t2h/must-gather-xr525" event={"ID":"04b92e95-3075-4f31-8824-ffe812e56c66","Type":"ContainerStarted","Data":"ed448b86e3e3d3000db4a8402a361d256eef1df511fea253fad07a312241147c"}
Sep 17 18:07:04.095811 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:04.095766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6t2h/must-gather-xr525" event={"ID":"04b92e95-3075-4f31-8824-ffe812e56c66","Type":"ContainerStarted","Data":"54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6"}
Sep 17 18:07:04.095811 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:04.095814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6t2h/must-gather-xr525" event={"ID":"04b92e95-3075-4f31-8824-ffe812e56c66","Type":"ContainerStarted","Data":"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"}
Sep 17 18:07:04.118002 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:04.117951    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k6t2h/must-gather-xr525" podStartSLOduration=1.094813283 podStartE2EDuration="7.11793497s" totalImagesPullingTime="6.023121687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:04.115251302 +0000 UTC m=+1562.983384268" watchObservedRunningTime="2026-09-17 18:07:04.11793497 +0000 UTC m=+1562.986067963"
Sep 17 18:07:27.212737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:27.212624    2581 generic.go:356] "Generic (PLEG): container finished" podID="04b92e95-3075-4f31-8824-ffe812e56c66" containerID="918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8" exitCode=0
Sep 17 18:07:27.212737 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:27.212697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6t2h/must-gather-xr525" event={"ID":"04b92e95-3075-4f31-8824-ffe812e56c66","Type":"ContainerDied","Data":"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"}
Sep 17 18:07:27.213157 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:27.213017    2581 scope.go:117] "RemoveContainer" containerID="918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"
Sep 17 18:07:27.939749 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:27.939723    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k6t2h_must-gather-xr525_04b92e95-3075-4f31-8824-ffe812e56c66/gather/0.log"
Sep 17 18:07:28.396241 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:28.396203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:28.577824 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:28.577782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:28.681423 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:28.681339    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:31.537896 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:31.537862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5rspk_1e889876-aac5-4dda-ad6d-cf30f00dc6ab/global-pull-secret-syncer/0.log"
Sep 17 18:07:31.691270 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:31.691235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log"
Sep 17 18:07:31.792198 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:31.792129    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-156.ec2.internal_500d1624d8f41e3c56732fe6161a664f/haproxy/0.log"
Sep 17 18:07:33.382042 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.382010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-k6t2h/must-gather-xr525"]
Sep 17 18:07:33.382579 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.382222    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-k6t2h/must-gather-xr525" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="copy" containerID="cri-o://54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6" gracePeriod=2
Sep 17 18:07:33.387041 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.387010    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-k6t2h/must-gather-xr525"]
Sep 17 18:07:33.608319 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.608297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k6t2h_must-gather-xr525_04b92e95-3075-4f31-8824-ffe812e56c66/copy/0.log"
Sep 17 18:07:33.609904 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.609886    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:07:33.612592 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:07:33.612561    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xr525\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-k6t2h\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:07:33.721449 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.721367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h227\" (UniqueName: \"kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227\") pod \"04b92e95-3075-4f31-8824-ffe812e56c66\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") "
Sep 17 18:07:33.721449 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.721425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output\") pod \"04b92e95-3075-4f31-8824-ffe812e56c66\" (UID: \"04b92e95-3075-4f31-8824-ffe812e56c66\") "
Sep 17 18:07:33.723359 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.723328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "04b92e95-3075-4f31-8824-ffe812e56c66" (UID: "04b92e95-3075-4f31-8824-ffe812e56c66"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:33.723497 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.723476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227" (OuterVolumeSpecName: "kube-api-access-7h227") pod "04b92e95-3075-4f31-8824-ffe812e56c66" (UID: "04b92e95-3075-4f31-8824-ffe812e56c66"). InnerVolumeSpecName "kube-api-access-7h227". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:33.822670 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.822625    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h227\" (UniqueName: \"kubernetes.io/projected/04b92e95-3075-4f31-8824-ffe812e56c66-kube-api-access-7h227\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:07:33.822670 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:33.822668    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b92e95-3075-4f31-8824-ffe812e56c66-must-gather-output\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\""
Sep 17 18:07:34.240517 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.240489    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k6t2h_must-gather-xr525_04b92e95-3075-4f31-8824-ffe812e56c66/copy/0.log"
Sep 17 18:07:34.240905 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.240879    2581 generic.go:356] "Generic (PLEG): container finished" podID="04b92e95-3075-4f31-8824-ffe812e56c66" containerID="54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6" exitCode=143
Sep 17 18:07:34.240976 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.240926    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:07:34.240976 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.240947    2581 scope.go:117] "RemoveContainer" containerID="54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6"
Sep 17 18:07:34.243806 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:07:34.243779    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xr525\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-k6t2h\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:07:34.249053 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.249035    2581 scope.go:117] "RemoveContainer" containerID="918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"
Sep 17 18:07:34.251155 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:07:34.251125    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xr525\" is forbidden: User \"system:node:ip-10-0-134-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-k6t2h\": no relationship found between node 'ip-10-0-134-156.ec2.internal' and this object" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" pod="openshift-must-gather-k6t2h/must-gather-xr525"
Sep 17 18:07:34.260086 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.260065    2581 scope.go:117] "RemoveContainer" containerID="54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6"
Sep 17 18:07:34.260320 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:07:34.260299    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6\": container with ID starting with 54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6 not found: ID does not exist" containerID="54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6"
Sep 17 18:07:34.260373 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.260329    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6"} err="failed to get container status \"54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6\": rpc error: code = NotFound desc = could not find container \"54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6\": container with ID starting with 54101a6c2485f343a4e23fa373f558053438fc5d44cb67a101881cd5e6d9baa6 not found: ID does not exist"
Sep 17 18:07:34.260373 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.260345    2581 scope.go:117] "RemoveContainer" containerID="918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"
Sep 17 18:07:34.260573 ip-10-0-134-156 kubenswrapper[2581]: E0917 18:07:34.260551    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8\": container with ID starting with 918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8 not found: ID does not exist" containerID="918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"
Sep 17 18:07:34.260620 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.260578    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8"} err="failed to get container status \"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8\": rpc error: code = NotFound desc = could not find container \"918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8\": container with ID starting with 918cfb6a861493ef449373d6b1c7dc7fa1fc5a7d9d63e522b3ff8c11434887e8 not found: ID does not exist"
Sep 17 18:07:34.362837 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.362811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:34.428698 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:34.428665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lpl5h_4157521a-db9f-46a3-922e-afd22e39edb3/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:35.360716 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:35.360687    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" path="/var/lib/kubelet/pods/04b92e95-3075-4f31-8824-ffe812e56c66/volumes"
Sep 17 18:07:36.120268 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.120240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/alertmanager/0.log"
Sep 17 18:07:36.153569 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.153541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/config-reloader/0.log"
Sep 17 18:07:36.187741 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.187720    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/kube-rbac-proxy-web/0.log"
Sep 17 18:07:36.220386 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.220364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/kube-rbac-proxy/0.log"
Sep 17 18:07:36.251048 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.251030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/kube-rbac-proxy-metric/0.log"
Sep 17 18:07:36.283201 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.283178    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/prom-label-proxy/0.log"
Sep 17 18:07:36.317343 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.317323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_203923fa-ab2e-4747-852c-f89be5609ddc/init-config-reloader/0.log"
Sep 17 18:07:36.407960 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.407892    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4bzn2_0565db71-00dd-4406-9807-f6be68ad05e5/kube-state-metrics/0.log"
Sep 17 18:07:36.438722 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.438705    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4bzn2_0565db71-00dd-4406-9807-f6be68ad05e5/kube-rbac-proxy-main/0.log"
Sep 17 18:07:36.479652 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.479629    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4bzn2_0565db71-00dd-4406-9807-f6be68ad05e5/kube-rbac-proxy-self/0.log"
Sep 17 18:07:36.519767 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.519739    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64c4d86f84-ssf8j_7f19eedf-1564-42a6-b7df-17521aa1a277/metrics-server/0.log"
Sep 17 18:07:36.595836 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.595814    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hjj5_0484b5ea-a518-4abf-b94a-22837c9d2753/node-exporter/0.log"
Sep 17 18:07:36.625940 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.625916    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hjj5_0484b5ea-a518-4abf-b94a-22837c9d2753/kube-rbac-proxy/0.log"
Sep 17 18:07:36.661239 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:36.661177    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hjj5_0484b5ea-a518-4abf-b94a-22837c9d2753/init-textfile/0.log"
Sep 17 18:07:37.346239 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:37.346215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5fd45c48-fq4vn_11cdb31e-25a8-4aba-bb14-b87df7083be9/telemeter-client/0.log"
Sep 17 18:07:37.379164 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:37.379139    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5fd45c48-fq4vn_11cdb31e-25a8-4aba-bb14-b87df7083be9/reload/0.log"
Sep 17 18:07:37.415933 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:37.415913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c5fd45c48-fq4vn_11cdb31e-25a8-4aba-bb14-b87df7083be9/kube-rbac-proxy/0.log"
Sep 17 18:07:39.509981 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.509951    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"]
Sep 17 18:07:39.510324 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510282    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="copy"
Sep 17 18:07:39.510324 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="copy"
Sep 17 18:07:39.510324 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510303    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="gather"
Sep 17 18:07:39.510324 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="gather"
Sep 17 18:07:39.510485 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510359    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="gather"
Sep 17 18:07:39.510485 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.510370    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b92e95-3075-4f31-8824-ffe812e56c66" containerName="copy"
Sep 17 18:07:39.514040 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.514023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.520539 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.520517    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66kd5\"/\"kube-root-ca.crt\""
Sep 17 18:07:39.521375 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.521355    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-66kd5\"/\"default-dockercfg-frhg2\""
Sep 17 18:07:39.521467 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.521355    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66kd5\"/\"openshift-service-ca.crt\""
Sep 17 18:07:39.533858 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.533840    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"]
Sep 17 18:07:39.668953 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.668919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-podres\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.668953 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.668951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-lib-modules\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.669187 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.668968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7kg\" (UniqueName: \"kubernetes.io/projected/c20573fb-46c8-4073-bb82-67e900d76c97-kube-api-access-lf7kg\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.669187 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.668991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-proc\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.669187 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.669059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-sys\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770471 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-sys\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770471 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-podres\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770471 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-lib-modules\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7kg\" (UniqueName: \"kubernetes.io/projected/c20573fb-46c8-4073-bb82-67e900d76c97-kube-api-access-lf7kg\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-proc\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-podres\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-proc\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-sys\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.770743 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.770631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c20573fb-46c8-4073-bb82-67e900d76c97-lib-modules\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.779988 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.779968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7kg\" (UniqueName: \"kubernetes.io/projected/c20573fb-46c8-4073-bb82-67e900d76c97-kube-api-access-lf7kg\") pod \"perf-node-gather-daemonset-ccgl7\" (UID: \"c20573fb-46c8-4073-bb82-67e900d76c97\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.824062 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.824026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:39.878634 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:39.878607    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9fd7587ff-8plxx_1bda909b-9cfe-4cfb-ab16-8e4c913b035a/console/0.log"
Sep 17 18:07:40.093752 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:40.093727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"]
Sep 17 18:07:40.095488 ip-10-0-134-156 kubenswrapper[2581]: W0917 18:07:40.095462    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc20573fb_46c8_4073_bb82_67e900d76c97.slice/crio-ac44720ef718259e50b95ce398aed40865e23996d66b20ac35504c1dc6bb775d WatchSource:0}: Error finding container ac44720ef718259e50b95ce398aed40865e23996d66b20ac35504c1dc6bb775d: Status 404 returned error can't find the container with id ac44720ef718259e50b95ce398aed40865e23996d66b20ac35504c1dc6bb775d
Sep 17 18:07:40.266600 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:40.266569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7" event={"ID":"c20573fb-46c8-4073-bb82-67e900d76c97","Type":"ContainerStarted","Data":"8e5e1adc1d6a69d3aebecbe146af566c943a927a074116cfa3551e0766e3067a"}
Sep 17 18:07:40.266758 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:40.266619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7" event={"ID":"c20573fb-46c8-4073-bb82-67e900d76c97","Type":"ContainerStarted","Data":"ac44720ef718259e50b95ce398aed40865e23996d66b20ac35504c1dc6bb775d"}
Sep 17 18:07:40.266758 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:40.266670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:40.285852 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:40.285811    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7" podStartSLOduration=1.285797767 podStartE2EDuration="1.285797767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:40.284833768 +0000 UTC m=+1599.152966731" watchObservedRunningTime="2026-09-17 18:07:40.285797767 +0000 UTC m=+1599.153930730"
Sep 17 18:07:41.583788 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:41.583759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tw6cx_587489eb-4e77-4bb2-b22a-8ed06399d34b/dns/0.log"
Sep 17 18:07:41.606888 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:41.606867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tw6cx_587489eb-4e77-4bb2-b22a-8ed06399d34b/kube-rbac-proxy/0.log"
Sep 17 18:07:41.661810 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:41.661788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhwgg_c3029b88-7089-4632-bd93-a6323afb9b49/dns-node-resolver/0.log"
Sep 17 18:07:42.137830 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:42.137803    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-586df9fd45-769kt_64179cb8-717e-4504-9007-45b11c1d9a2d/registry/0.log"
Sep 17 18:07:42.187147 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:42.187123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gjbf_47daedcd-8541-40ae-847e-7b786c4a982b/node-ca/0.log"
Sep 17 18:07:43.420404 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:43.420378    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cp6jw_2c9ad593-3c0f-4a10-aa7f-17972148edd9/serve-healthcheck-canary/0.log"
Sep 17 18:07:43.913046 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:43.913004    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzz5j_89d826d4-b800-4639-9e65-e6678e7fd081/kube-rbac-proxy/0.log"
Sep 17 18:07:43.941057 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:43.941030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzz5j_89d826d4-b800-4639-9e65-e6678e7fd081/exporter/0.log"
Sep 17 18:07:43.972259 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:43.972232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzz5j_89d826d4-b800-4639-9e65-e6678e7fd081/extractor/0.log"
Sep 17 18:07:46.280793 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:46.280765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-ccgl7"
Sep 17 18:07:46.742696 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:46.742596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-thn7f_d997268d-93a5-45e9-94a7-83dda54ad5a6/seaweedfs/0.log"
Sep 17 18:07:51.470823 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:51.470795    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-m84bj_77002440-0a3c-46bc-b7c9-d26de142d234/migrator/0.log"
Sep 17 18:07:51.496715 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:51.496686    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-m84bj_77002440-0a3c-46bc-b7c9-d26de142d234/graceful-termination/0.log"
Sep 17 18:07:53.143377 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.143351    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:07:53.167213 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.167192    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/egress-router-binary-copy/0.log"
Sep 17 18:07:53.190350 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.190332    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/cni-plugins/0.log"
Sep 17 18:07:53.216621 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.216603    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/bond-cni-plugin/0.log"
Sep 17 18:07:53.240625 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.240605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/routeoverride-cni/0.log"
Sep 17 18:07:53.268180 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.268164    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/whereabouts-cni-bincopy/0.log"
Sep 17 18:07:53.294610 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.294587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmzvx_2a4a871b-b563-4ba1-b00e-d0687edf4d03/whereabouts-cni/0.log"
Sep 17 18:07:53.597769 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.597740    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vsfpn_ad99e04b-e33b-432d-bb24-dc9da16c2f3f/kube-multus/0.log"
Sep 17 18:07:53.665439 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.665415    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4cbwc_8215753a-c993-4f2a-82d1-756ab30df519/network-metrics-daemon/0.log"
Sep 17 18:07:53.693466 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:53.693437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4cbwc_8215753a-c993-4f2a-82d1-756ab30df519/kube-rbac-proxy/0.log"
Sep 17 18:07:54.831274 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.831241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-controller/0.log"
Sep 17 18:07:54.888128 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.888097    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/1.log"
Sep 17 18:07:54.888128 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.888114    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovn-acl-logging/0.log"
Sep 17 18:07:54.907975 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.907947    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/kube-rbac-proxy-node/0.log"
Sep 17 18:07:54.931788 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.931764    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:07:54.960762 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.960744    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/northd/0.log"
Sep 17 18:07:54.998104 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:54.998083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/nbdb/0.log"
Sep 17 18:07:55.027676 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:55.027634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/sbdb/0.log"
Sep 17 18:07:55.068703 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:55.068681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g6ksl_db7e86f5-234c-409d-96c1-a57c43db924e/ovnkube-controller/0.log"
Sep 17 18:07:56.393190 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:56.393163    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lfzxt_e8fa2840-f25a-4ab4-b208-4a54ae5dfafc/network-check-target-container/0.log"
Sep 17 18:07:57.587550 ip-10-0-134-156 kubenswrapper[2581]: I0917 18:07:57.587521    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jvzls_1ff9e614-f637-4494-a951-05bc50d222e2/iptables-alerter/0.log"